2024-03-24T04:03:09.237 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T04:03:09.237 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847 --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/orig.config.yaml 2024-03-24T04:03:09.272 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T04:03:09.353 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617847' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOdWTIXrRM+iCeSvS9IDw2aoJZ06gm3sWkuonouXzH6r5kkQbWIfcve8AKmLA76lBtIMmJOhOUA/gCU18dN0l+o= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqLqfO5Av72MO/1zBNgmF7F6/oz+SmYEXyi1RqQphw95A71H44jESKZrmdCF624G49Bqj9/emWZXbQP9EtOkcs= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-03-24T04:03:09.353 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T04:03:09.353 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T04:03:09.353 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T04:03:09.355 INFO:teuthology.task.internal:Checking packages... 2024-03-24T04:03:09.377 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T04:03:09.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T04:03:09.377 INFO:teuthology.packaging:ref: None 2024-03-24T04:03:09.377 INFO:teuthology.packaging:tag: None 2024-03-24T04:03:09.377 INFO:teuthology.packaging:branch: squid 2024-03-24T04:03:09.377 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:03:09.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T04:03:09.611 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T04:03:09.612 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T04:03:09.619 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T04:03:09.619 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T04:03:09.628 INFO:teuthology.task.internal:Saving configuration 2024-03-24T04:03:09.641 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T04:03:09.646 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T04:03:09.671 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 03:58:06.248333', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOdWTIXrRM+iCeSvS9IDw2aoJZ06gm3sWkuonouXzH6r5kkQbWIfcve8AKmLA76lBtIMmJOhOUA/gCU18dN0l+o='} 2024-03-24T04:03:09.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 03:58:06.249706', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqLqfO5Av72MO/1zBNgmF7F6/oz+SmYEXyi1RqQphw95A71H44jESKZrmdCF624G49Bqj9/emWZXbQP9EtOkcs='} 2024-03-24T04:03:09.696 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T04:03:09.702 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-24T04:03:09.703 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-24T04:03:09.703 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T04:03:09.769 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f918606cf70>, signals=[15]) 2024-03-24T04:03:09.770 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T04:03:09.776 INFO:teuthology.task.internal:Opening connections... 2024-03-24T04:03:09.776 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:03:09.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:03:09.880 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T04:03:09.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:03:09.968 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T04:03:09.974 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-24T04:03:09.993 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-24T04:03:09.993 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-24T04:03:10.049 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-24T04:03:10.049 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T04:03:10.050 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T04:03:10.051 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-24T04:03:10.072 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-24T04:03:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-24T04:03:10.088 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-24T04:03:10.144 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T04:03:10.145 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T04:03:10.146 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-24T04:03:10.163 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T04:03:10.171 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T04:03:10.178 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T04:03:10.178 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-24T04:03:10.180 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-24T04:03:10.200 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T04:03:10.206 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T04:03:10.206 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-24T04:03:10.238 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-24T04:03:10.270 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T04:03:10.324 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T04:03:10.338 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T04:03:10.338 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T04:03:10.338 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T04:03:10.339 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T04:03:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:03:10.353 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-24T04:03:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:03:10.355 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T04:03:10.409 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-24T04:03:10.409 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-24T04:03:10.409 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-24T04:03:10.413 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-24T04:03:10.413 INFO:teuthology.task.kernel:Running kernel on smithi022: 5.14.0-361.el9.x86_64 2024-03-24T04:03:10.413 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2024-03-24T04:03:12.578 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T04:03:12.616 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T04:03:14.046 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-24T04:03:14.066 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-24T04:03:14.890 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-24T04:03:15.280 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-24T04:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-24T04:03:17.248 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-24T04:03:24.229 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-24T04:03:25.738 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-24T04:03:29.807 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 674 kB/s | 6.1 MB 00:09 2024-03-24T04:03:31.475 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-24T04:03:32.559 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-24T04:03:35.424 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 21 MB 00:09 2024-03-24T04:03:37.668 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-24T04:03:39.825 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T04:03:39.825 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T04:03:39.927 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:03:39.939 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 76 M 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 123 M 2024-03-24T04:03:39.940 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T04:03:40.102 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 372 kB/s | 44 kB 00:00 2024-03-24T04:03:40.463 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-24T04:03:40.886 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-24T04:03:41.421 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 21 MB/s | 27 MB 00:01 2024-03-24T04:03:41.804 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 18 MB/s | 33 MB 00:01 2024-03-24T04:03:41.804 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T04:03:41.804 INFO:teuthology.orchestra.run.smithi022.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-24T04:03:42.494 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T04:03:42.495 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T04:03:42.500 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T04:03:42.581 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T04:03:42.593 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-24T04:03:42.594 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:03:42.594 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 76 M 2024-03-24T04:03:42.594 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-24T04:03:42.594 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-24T04:03:42.946 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T04:03:42.946 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T04:03:43.498 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-24T04:03:43.983 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T04:03:43.983 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T04:03:46.608 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T04:03:47.251 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T04:03:47.337 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:03:50.218 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:03:50.348 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:03:51.514 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-modules-core-5.14.0-430.el9.x86_6 3.4 MB/s | 27 MB 00:08 2024-03-24T04:03:55.449 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:03:55.492 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:00.634 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:03.096 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 755 kB/s | 15 MB 00:20 2024-03-24T04:04:25.443 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 805 kB/s | 33 MB 00:42 2024-03-24T04:04:25.444 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-24T04:04:25.444 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.8 MB/s | 76 MB 00:42 2024-03-24T04:04:26.104 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-24T04:04:26.491 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-24T04:04:26.491 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-24T04:04:27.500 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-24T04:04:27.500 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-24T04:04:29.969 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-24T04:04:30.588 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T04:04:30.668 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:04:31.468 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:31.468 INFO:teuthology.orchestra.run.smithi022.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T04:04:31.468 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:04:31.515 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:31.901 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:31.901 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T04:04:31.901 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:04:31.901 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:04:32.429 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:04:32.430 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T04:04:32.777 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T04:04:32.877 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.878 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T04:04:32.878 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T04:04:32.878 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T04:04:32.878 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T04:04:32.878 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-24T04:04:32.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T04:04:32.878 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T04:04:32.960 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T04:04:32.961 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T04:04:32.961 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-24T04:04:32.998 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-24T04:04:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:04:33.000 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T04:04:33.000 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T04:04:33.000 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 5.14.0-430.el9.x86_64 2024-03-24T04:04:33.000 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T04:04:33.430 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:04:33.563 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:04:33.619 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-24T04:04:35.008 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T04:04:35.040 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-24T04:04:35.043 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-24T04:04:35.059 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.IXX9uTA19d 2024-03-24T04:04:35.059 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IXX9uTA19d 2024-03-24T04:04:35.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.IXX9uTA19d 2024-03-24T04:04:35.279 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.IXX9uTA19d is 6KB 2024-03-24T04:04:35.329 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.IXX9uTA19d 2024-03-24T04:04:35.344 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T04:04:35.413 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T04:04:35.413 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T04:04:35.413 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T04:04:35.413 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T04:04:35.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T04:04:35.565 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-24T04:04:38.290 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:04:38.429 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:04:43.201 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:05:05.568 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-24T04:05:05.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T04:05:05.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:05:12.383 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:05:12.384 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T04:05:12.384 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:05:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:05:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:05:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T04:05:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T04:05:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:05:13.205 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-24T04:05:13.516 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T04:05:13.637 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.637 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T04:05:13.637 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T04:05:13.637 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T04:05:13.637 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T04:05:13.637 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-24T04:05:13.637 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T04:05:13.638 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T04:05:13.717 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T04:05:13.718 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T04:05:13.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-24T04:05:13.762 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-24T04:05:13.763 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:05:13.764 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T04:05:13.765 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T04:05:13.765 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-430.el9.x86_64 2024-03-24T04:05:13.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T04:05:14.411 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-24T04:05:15.858 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T04:05:15.904 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-24T04:05:15.906 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-24T04:05:15.921 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.kbpgnmLxA2 2024-03-24T04:05:15.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kbpgnmLxA2 2024-03-24T04:05:15.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.kbpgnmLxA2 2024-03-24T04:05:16.131 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.kbpgnmLxA2 is 6KB 2024-03-24T04:05:16.179 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.kbpgnmLxA2 2024-03-24T04:05:16.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T04:05:16.257 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T04:05:16.258 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T04:05:16.258 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T04:05:16.258 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T04:05:16.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T04:05:16.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-24T04:05:46.435 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-24T04:05:46.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:05:46.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:06:05.591 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T04:06:14.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T04:06:14.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:06:14.930 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T04:06:15.336 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T04:06:15.337 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T04:06:15.337 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T04:06:15.350 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-430.el9.x86_64 2024-03-24T04:06:15.351 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T04:06:15.351 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T04:06:15.351 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T04:06:16.352 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-24T04:06:16.352 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T04:06:16.446 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2024-03-24T04:06:16.488 DEBUG:teuthology.parallel:result is None 2024-03-24T04:06:20.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-24T04:06:29.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:06:29.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:06:48.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-24T04:07:00.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:07:00.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:07:00.667 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-24T04:07:01.087 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:07:01.087 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T04:07:01.088 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-24T04:07:01.102 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-430.el9.x86_64 2024-03-24T04:07:01.102 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T04:07:01.102 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T04:07:01.102 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T04:07:02.103 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-24T04:07:02.103 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T04:07:02.158 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-24T04:07:02.197 DEBUG:teuthology.parallel:result is None 2024-03-24T04:07:02.204 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T04:07:02.212 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T04:07:02.212 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T04:07:02.215 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T04:07:02.230 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T04:07:02.237 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T04:07:02.248 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T04:07:02.248 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T04:07:02.273 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T04:07:02.297 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T04:07:02.303 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T04:07:02.304 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T04:07:02.337 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T04:07:02.374 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T04:07:02.375 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T04:07:02.388 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T04:07:02.389 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T04:07:02.390 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T04:07:02.397 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T04:07:02.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T04:07:02.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T04:07:02.461 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T04:07:02.470 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T04:07:02.470 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T04:07:02.504 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T04:07:02.518 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T04:07:02.598 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T04:07:02.649 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T04:07:02.731 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T04:07:02.775 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:07:02.776 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T04:07:02.843 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T04:07:02.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T04:07:02.939 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T04:07:03.017 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T04:07:03.064 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:07:03.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T04:07:03.136 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-24T04:07:03.138 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-24T04:07:03.188 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T04:07:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T04:07:03.590 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T04:07:03.598 INFO:teuthology.task.internal:Starting timer... 2024-03-24T04:07:03.598 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T04:07:03.609 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T04:07:03.618 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T04:07:03.619 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-24T04:07:03.674 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-24T04:07:03.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-24T04:07:03.747 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-24T04:07:03.749 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T04:07:03.749 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-24T04:07:03.776 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-24T04:07:03.777 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-24T04:07:03.799 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-24T04:07:03.799 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive'} 2024-03-24T04:07:03.799 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T04:07:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:07:03.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T04:07:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:07:03.872 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T04:07:03.872 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-24T04:07:03.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-24T04:07:03.959 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T04:07:03.969 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T04:07:03.974 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T04:07:04.090 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T04:07:04.103 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T04:07:04.105 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T04:11:15.274 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2024-03-24T04:11:15.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T04:11:15.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:11:15.365 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T04:11:15.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T04:11:15.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:11:15.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T04:11:15.530 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-24T04:11:15.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T04:11:15.606 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T04:11:15.617 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T04:11:15.617 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T04:11:15.618 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T04:11:15.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T04:11:15.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T04:11:15.648 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T04:11:15.665 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T04:11:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T04:11:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T04:11:15.694 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-24T04:11:15.708 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-24T04:11:15.722 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-24T04:11:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T04:11:15.735 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-24T04:11:15.740 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T04:11:15.751 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T04:11:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T04:11:15.791 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.794 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T04:11:15.829 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T04:11:15.829 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T04:11:15.830 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.830 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.830 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.830 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:11:15.830 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T04:11:15.941 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T04:11:15.941 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-24T04:11:15.941 INFO:tasks.cephadm:Cluster fsid is 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:15.941 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T04:11:15.942 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-24T04:11:15.942 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi022': '172.21.15.22', 'mon.smithi116': '172.21.15.116'} 2024-03-24T04:11:15.942 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T04:11:15.942 DEBUG:teuthology.orchestra.run.smithi022:> sudo hostname $(hostname -s) 2024-03-24T04:11:15.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-24T04:11:15.998 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-24T04:11:15.999 DEBUG:teuthology.orchestra.run.smithi022:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 04:11 /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.297 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.505 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 04:11 /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.506 DEBUG:teuthology.orchestra.run.smithi022:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.522 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T04:11:16.545 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-24T04:11:16.545 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T04:11:16.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T04:11:17.473 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T04:11:17.479 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T04:11:36.665 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:11:36.665 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout: "repo_digests": [ 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T04:11:36.666 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:11:38.628 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T04:11:38.628 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T04:11:38.628 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T04:11:38.629 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-24T04:11:38.629 INFO:teuthology.orchestra.run.smithi116.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T04:11:38.629 INFO:teuthology.orchestra.run.smithi116.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T04:11:38.629 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T04:11:38.629 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T04:11:38.646 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph 2024-03-24T04:11:38.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-24T04:11:38.710 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /etc/ceph 2024-03-24T04:11:38.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-24T04:11:38.778 INFO:tasks.cephadm:Writing seed config... 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T04:11:38.779 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T04:11:38.780 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T04:11:38.780 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T04:11:38.780 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:11:38.780 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T04:11:38.808 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8f0be64c-e994-11ee-95cd-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T04:11:38.808 DEBUG:teuthology.orchestra.run.smithi022:mon.smithi022> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service 2024-03-24T04:11:38.851 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T04:11:38.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.22 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:11:39.055 INFO:teuthology.orchestra.run.smithi022.stderr:-------------------------------------------------------------------------------- 2024-03-24T04:11:39.056 INFO:teuthology.orchestra.run.smithi022.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '8f0be64c-e994-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.22', '--skip-admin-label'] 2024-03-24T04:11:39.079 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 4.9.3-dev 2024-03-24T04:11:39.081 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying podman|docker is present... 2024-03-24T04:11:39.102 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 4.9.3-dev 2024-03-24T04:11:39.104 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying lvm2 is present... 2024-03-24T04:11:39.104 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying time synchronization is in place... 2024-03-24T04:11:39.111 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T04:11:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:11:39.123 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: enabled 2024-03-24T04:11:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: active 2024-03-24T04:11:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:Unit chronyd.service is enabled and running 2024-03-24T04:11:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:Repeating the final host check... 2024-03-24T04:11:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 4.9.3-dev 2024-03-24T04:11:39.152 INFO:teuthology.orchestra.run.smithi022.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-24T04:11:39.152 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl is present 2024-03-24T04:11:39.153 INFO:teuthology.orchestra.run.smithi022.stderr:lvcreate is present 2024-03-24T04:11:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T04:11:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:11:39.170 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: enabled 2024-03-24T04:11:39.176 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: active 2024-03-24T04:11:39.177 INFO:teuthology.orchestra.run.smithi022.stderr:Unit chronyd.service is enabled and running 2024-03-24T04:11:39.177 INFO:teuthology.orchestra.run.smithi022.stderr:Host looks OK 2024-03-24T04:11:39.177 INFO:teuthology.orchestra.run.smithi022.stderr:Cluster fsid: 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:39.177 INFO:teuthology.orchestra.run.smithi022.stderr:Acquiring lock 139977151651216 on /run/cephadm/8f0be64c-e994-11ee-95cd-87774f69a715.lock 2024-03-24T04:11:39.177 INFO:teuthology.orchestra.run.smithi022.stderr:Lock 139977151651216 acquired on /run/cephadm/8f0be64c-e994-11ee-95cd-87774f69a715.lock 2024-03-24T04:11:39.178 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying IP 172.21.15.22 port 3300 ... 2024-03-24T04:11:39.178 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying IP 172.21.15.22 port 6789 ... 2024-03-24T04:11:39.178 INFO:teuthology.orchestra.run.smithi022.stderr:Base mon IP is 172.21.15.22, final addrv is [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T04:11:39.180 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.22 metric 100 2024-03-24T04:11:39.181 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.22 metric 100 2024-03-24T04:11:39.183 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T04:11:39.183 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T04:11:39.183 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T04:11:39.185 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T04:11:39.186 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-24T04:11:39.186 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T04:11:39.186 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T04:11:39.186 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T04:11:39.186 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T04:11:39.187 INFO:teuthology.orchestra.run.smithi022.stderr:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2024-03-24T04:11:39.187 INFO:teuthology.orchestra.run.smithi022.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T04:11:39.187 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T04:11:39.290 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Getting image source signatures 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-24T04:11:39.715 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-24T04:11:39.716 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T04:11:39.868 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Writing manifest to image destination 2024-03-24T04:11:40.021 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T04:11:40.396 INFO:teuthology.orchestra.run.smithi022.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T04:11:40.970 INFO:teuthology.orchestra.run.smithi022.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T04:11:40.971 INFO:teuthology.orchestra.run.smithi022.stderr:Extracting ceph user uid/gid from container image... 2024-03-24T04:11:41.403 INFO:teuthology.orchestra.run.smithi022.stderr:stat: 167 167 2024-03-24T04:11:41.910 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial keys... 2024-03-24T04:11:42.433 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQD+p/9lhjzuFBAAHKatf74WrooNzwcXuUO+Vw== 2024-03-24T04:11:43.331 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQD/p/9ldIC4DhAAGEopUYcr981SX+I6MVducA== 2024-03-24T04:11:44.486 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQAAqP9l8n32FxAAt+I732W9Clo0azuTxiYOyg== 2024-03-24T04:11:44.960 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial monmap... 2024-03-24T04:11:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T04:11:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-24T04:11:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T04:11:45.856 INFO:teuthology.orchestra.run.smithi022.stderr:monmaptool for smithi022 [v2:172.21.15.22:3300,v1:172.21.15.22:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T04:11:45.856 INFO:teuthology.orchestra.run.smithi022.stderr:setting min_mon_release = octopus 2024-03-24T04:11:45.856 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: set fsid to 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:45.856 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T04:11:45.857 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:11:45.857 INFO:teuthology.orchestra.run.smithi022.stderr:Creating mon... 2024-03-24T04:11:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:create mon.smithi022 on 2024-03-24T04:11:46.976 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T04:11:47.119 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target. 2024-03-24T04:11:47.119 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target. 2024-03-24T04:11:47.406 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to reset failed state of unit ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service: Unit ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service not loaded. 2024-03-24T04:11:47.413 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target.wants/ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service → /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715@.service. 2024-03-24T04:11:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2024-03-24T04:11:48.023 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:11:48.024 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2024-03-24T04:11:48.024 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T04:11:48.025 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mon to start... 2024-03-24T04:11:48.025 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mon... 2024-03-24T04:11:48.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:48 smithi022 systemd[1]: Started Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:11:48.544 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: cluster: 2024-03-24T04:11:48.544 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: id: 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:48.544 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: services: 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi022 (age 0.541296s) 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: data: 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-24T04:11:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-24T04:11:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: pgs: 2024-03-24T04:11:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:49.013 INFO:teuthology.orchestra.run.smithi022.stderr:mon is available 2024-03-24T04:11:49.013 INFO:teuthology.orchestra.run.smithi022.stderr:Assimilating anything we can from ceph.conf... 2024-03-24T04:11:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: mon.smithi022 is new leader, mons smithi022 in quorum (ranks 0) 2024-03-24T04:11:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: monmap e1: 1 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0]} 2024-03-24T04:11:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: fsmap 2024-03-24T04:11:49.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T04:11:49.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: mgrmap e1: no daemons active 2024-03-24T04:11:49.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:49 smithi022 ceph-mon[24420]: from='client.? 172.21.15.22:0/311654589' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [global] 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: fsid = 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T04:11:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [mgr] 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [osd] 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T04:11:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T04:11:50.027 INFO:teuthology.orchestra.run.smithi022.stderr:Generating new minimal ceph.conf... 2024-03-24T04:11:50.134 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:50 smithi022 ceph-mon[24420]: from='client.? 172.21.15.22:0/1057446784' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T04:11:50.134 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:50 smithi022 ceph-mon[24420]: from='client.? 172.21.15.22:0/1057446784' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T04:11:50.983 INFO:teuthology.orchestra.run.smithi022.stderr:Restarting the monitor... 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 ceph-mon[24420]: from='client.? 172.21.15.22:0/1741089727' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 systemd[1]: Stopping Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 bash[24636]: Error: no container with name or ID "ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon.smithi022" found: no such container 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[24416]: 2024-03-24T04:11:51.141+0000 7f870401d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi022 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[24416]: 2024-03-24T04:11:51.141+0000 7f870401d700 -1 mon.smithi022@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T04:11:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 podman[24646]: 2024-03-24 04:11:51.162973341 +0000 UTC m=+0.114260125 container died 16c07a3041e95a3f80ea2f0b34d4369ff0279e0331fb6d213fade5f531a3726d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, version=8, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T04:11:51.767 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 podman[24646]: 2024-03-24 04:11:51.533731628 +0000 UTC m=+0.485018419 container cleanup 16c07a3041e95a3f80ea2f0b34d4369ff0279e0331fb6d213fade5f531a3726d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=centos-stream, GIT_CLEAN=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-24T04:11:51.767 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 bash[24646]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022 2024-03-24T04:11:51.768 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 bash[24673]: Error: no container with name or ID "ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon.smithi022" found: no such container 2024-03-24T04:11:52.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 podman[24660]: 2024-03-24 04:11:51.767902215 +0000 UTC m=+0.608026157 container remove 16c07a3041e95a3f80ea2f0b34d4369ff0279e0331fb6d213fade5f531a3726d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T04:11:52.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service: Deactivated successfully. 2024-03-24T04:11:52.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 systemd[1]: Stopped Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:11:52.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 systemd[1]: Starting Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:11:52.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:51 smithi022 podman[24737]: 2024-03-24 04:11:51.985451282 +0000 UTC m=+0.094986577 container create 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream) 2024-03-24T04:11:52.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 podman[24737]: 2024-03-24 04:11:51.908710404 +0000 UTC m=+0.018245637 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-24T04:11:52.274 INFO:teuthology.orchestra.run.smithi022.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 podman[24737]: 2024-03-24 04:11:52.187926199 +0000 UTC m=+0.297461444 container init 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, GIT_BRANCH=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, version=8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870) 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 podman[24737]: 2024-03-24 04:11:52.190967566 +0000 UTC m=+0.300502791 container start 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, version=8, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: pidfile_write: ignore empty --pid-file 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: load: jerasure load: lrc 2024-03-24T04:11:52.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: RocksDB version: 6.15.5 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Compile date Apr 18 2022 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: DB SUMMARY 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: DB Session ID: IL65QWI32ZIMWNUN1JTF 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: CURRENT file: CURRENT 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi022/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi022/store.db: 000010.log size: 82711 ; 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.error_if_exists: 0 2024-03-24T04:11:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.create_if_missing: 0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.paranoid_checks: 1 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.env: 0x559a8b4c6860 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.fs: Posix File System 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.info_log: 0x559a8c10dfa0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.statistics: (nil) 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.use_fsync: 0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_log_file_size: 0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T04:11:52.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_fallocate: 1 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.use_direct_reads: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.db_log_dir: 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi022/store.db 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T04:11:52.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.write_buffer_manager: 0x559a8c1fe330 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.unordered_write: 0 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T04:11:52.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.row_cache: None 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.wal_filter: None 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.preserve_deletes: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.two_write_queues: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.atomic_flush: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.log_readahead_size: 0 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T04:11:52.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_background_jobs: 2 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_background_compactions: -1 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_subcompactions: 1 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T04:11:52.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_open_files: -1 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_background_flushes: -1 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Compression algorithms supported: 2024-03-24T04:11:52.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kZSTD supported: 0 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kXpressCompression supported: 0 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kLZ4Compression supported: 1 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kBZip2Compression supported: 0 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kZlibCompression supported: 1 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: kSnappyCompression supported: 1 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000009 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.merge_operator: 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_filter: None 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T04:11:52.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a8c0d9d00) 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks: 1 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_top_level_index_and_filter: 1 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_type: 0 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_index_type: 0 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_shortening: 1 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: hash_index_allow_collision: 1 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: checksum: 1 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: no_block_cache: 0 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache: 0x559a8c144170 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_name: BinnedLRUCache 2024-03-24T04:11:52.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_options: 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: capacity : 536870912 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: num_shard_bits : 4 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: strict_capacity_limit : 0 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: high_pri_pool_ratio: 0.000 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_compressed: (nil) 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: persistent_cache: (nil) 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size: 4096 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size_deviation: 10 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_restart_interval: 16 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_block_restart_interval: 1 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: metadata_block_size: 4096 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: partition_filters: 0 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: use_delta_encoding: 1 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: whole_key_filtering: 1 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: verify_compression: 0 2024-03-24T04:11:52.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: read_amp_bytes_per_bit: 0 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: format_version: 4 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_index_compression: 1 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_align: 0 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression: NoCompression 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.num_levels: 7 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T04:11:52.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T04:11:52.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.arena_block_size: 4194304 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T04:11:52.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.table_properties_collectors: 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.inplace_update_support: 0 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T04:11:52.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.bloom_locality: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.max_successive_merges: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.ttl: 2592000 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.enable_blob_files: false 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.min_blob_size: 0 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T04:11:52.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253512216185, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253512216897, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79393, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77657, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71966, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711253512, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6d411d2c-f80f-48b0-917d-e903304978f2", "db_session_id": "IL65QWI32ZIMWNUN1JTF"}} 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253512217231, "job": 1, "event": "recovery_finished"} 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi022/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559a8c12aa80 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: DB pointer 0x559a8c13a000 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-24T04:11:52.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** DB Stats ** 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative compaction: 0.00 GB write, 25.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval compaction: 0.00 GB write, 25.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:11:52.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.5 0.00 0.00 1 0.000 0 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:11:52.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative compaction: 0.00 GB write, 25.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: starting mon.smithi022 rank 0 at public addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] at bind addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi022 fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???) e1 preinit fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).mds e1 new map 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).mds e1 print_map 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: e1 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: legacy client fscid: -1 2024-03-24T04:11:52.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: No filesystems configured 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 bash[24737]: 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 systemd[1]: Started Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mon.smithi022 is new leader, mons smithi022 in quorum (ranks 0) 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: monmap e1: 1 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0]} 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: fsmap 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T04:11:52.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:52 smithi022 ceph-mon[24756]: mgrmap e1: no daemons active 2024-03-24T04:11:53.325 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-24T04:11:53.325 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:11:53.325 INFO:teuthology.orchestra.run.smithi022.stderr:Creating mgr... 2024-03-24T04:11:53.325 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying port 9283 ... 2024-03-24T04:11:53.495 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to reset failed state of unit ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mgr.smithi022.ijpjvk.service: Unit ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mgr.smithi022.ijpjvk.service not loaded. 2024-03-24T04:11:53.502 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715.target.wants/ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mgr.smithi022.ijpjvk.service → /etc/systemd/system/ceph-8f0be64c-e994-11ee-95cd-87774f69a715@.service. 2024-03-24T04:11:53.980 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:53 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2437152656' entity='client.admin' 2024-03-24T04:11:54.195 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2024-03-24T04:11:54.203 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:11:54.204 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2024-03-24T04:11:54.204 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T04:11:54.213 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2024-03-24T04:11:54.221 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:11:54.222 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2024-03-24T04:11:54.222 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-24T04:11:54.222 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr to start... 2024-03-24T04:11:54.222 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr... 2024-03-24T04:11:54.785 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:11:54.785 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "8f0be64c-e994-11ee-95cd-87774f69a715", 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "smithi022" 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:11:54.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T04:11:54.787 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T04:11:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T04:11:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T04:11:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T04:11:54.791 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": false, 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2024-03-24T04:11:48.001160+0000", 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T04:11:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:11:55.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:11:54 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/893331149' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T04:11:55.364 INFO:teuthology.orchestra.run.smithi022.stderr:mgr not available, waiting (1/15)... 2024-03-24T04:12:00.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:12:00.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: mgrmap e2: smithi022.ijpjvk(active, starting, since 0.00264903s) 2024-03-24T04:12:00.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:12:00.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:12:00.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:12:00.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:12:00.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:12:00.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:00 smithi022 ceph-mon[24756]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:12:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:12:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:12:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='mgr.14100 172.21.15.22:0/1975353623' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:01.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3100407664' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T04:12:01.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:01 smithi022 ceph-mon[24756]: mgrmap e3: smithi022.ijpjvk(active, since 1.00541s) 2024-03-24T04:12:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:12:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "8f0be64c-e994-11ee-95cd-87774f69a715", 2024-03-24T04:12:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2024-03-24T04:12:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "smithi022" 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T04:12:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": false, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T04:12:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2024-03-24T04:11:48.001160+0000", 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T04:12:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:03 smithi022 ceph-mon[24756]: mgrmap e4: smithi022.ijpjvk(active, since 2s) 2024-03-24T04:12:07.365 INFO:teuthology.orchestra.run.smithi022.stderr:mgr not available, waiting (2/15)... 2024-03-24T04:12:10.130 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:12:10.130 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:10.130 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "8f0be64c-e994-11ee-95cd-87774f69a715", 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "smithi022" 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:10.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T04:12:10.133 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T04:12:10.133 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T04:12:10.133 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T04:12:10.137 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T04:12:10.137 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T04:12:10.137 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T04:12:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T04:12:10.139 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2024-03-24T04:11:48.001160+0000", 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2024-03-24T04:12:10.140 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T04:12:10.141 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:10.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:10 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/450237553' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T04:12:10.733 INFO:teuthology.orchestra.run.smithi022.stderr:mgr is available 2024-03-24T04:12:11.339 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [global] 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: fsid = 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [mgr] 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [osd] 2024-03-24T04:12:11.340 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T04:12:11.341 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T04:12:11.645 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:11 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2924873374' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T04:12:11.923 INFO:teuthology.orchestra.run.smithi022.stderr:Enabling cephadm module... 2024-03-24T04:12:13.526 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:13 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2486855667' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "active_name": "smithi022.ijpjvk", 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T04:12:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:14.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:14 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2486855667' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T04:12:14.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:14 smithi022 ceph-mon[24756]: mgrmap e5: smithi022.ijpjvk(active, since 13s) 2024-03-24T04:12:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for the mgr to restart... 2024-03-24T04:12:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr epoch 5... 2024-03-24T04:12:15.330 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:15 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2827189243' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T04:12:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-24T04:12:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T04:12:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: mgrmap e6: smithi022.ijpjvk(active, starting, since 0.0524875s) 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:12:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:12:20.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:20 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:12:20.892 INFO:teuthology.orchestra.run.smithi022.stderr:mgr epoch 5 is available 2024-03-24T04:12:20.892 INFO:teuthology.orchestra.run.smithi022.stderr:Setting orchestrator backend to cephadm... 2024-03-24T04:12:21.500 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:21 smithi022 ceph-mon[24756]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T04:12:21.500 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:21 smithi022 ceph-mon[24756]: mgrmap e7: smithi022.ijpjvk(active, since 1.0548s) 2024-03-24T04:12:22.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T04:12:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: mgrmap e8: smithi022.ijpjvk(active, since 2s) 2024-03-24T04:12:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:22 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:22.663 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: value unchanged 2024-03-24T04:12:23.238 INFO:teuthology.orchestra.run.smithi022.stderr:Generating ssh key... 2024-03-24T04:12:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:23 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:21] ENGINE Bus STARTING 2024-03-24T04:12:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:23 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:21] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:12:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:23 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:21] ENGINE Bus STARTED 2024-03-24T04:12:24.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:24 smithi022 ceph-mon[24756]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:24.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:24 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:24.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:24 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:25.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:25 smithi022 ceph-mon[24756]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:25.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:25 smithi022 ceph-mon[24756]: Generating ssh key... 2024-03-24T04:12:25.323 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWp7qvVef4Woo8fLPAibFjp8T2lKRtQtRw7sRcce3NquiF0dNizvtiaLo9JZnMkOKCxsR7uqa/aGTiUguxWcAWfWpFxEnT9Azin8o3qtF0BjhnWiBlvzaF5Z3uvRfupANwQSeo2f1CM4CoccUomdf1X4ksJfNYUCJ8/hbVAwpvj/cW1uof9+AdkwuT7r+N2ExfBmNpr0vV1iUiotUY4HHZTBPsrDmqXhN6WGVtBIUEslfnRlOOyxKgAGX7tgs88Qji5MvPFw9Gf/p6sC4ttT0tqcn+FgD1FUwdc/al5mf3MnLUkA2h3vDX2uA3wlMpQJtd1BHCEHGj7lCJ7cop1OxLX9ugZVZOx1TvT6jkhzH1xc2qlPZ5mAG0PwZkBq8K+vEuSpUquMtGiEnsMEKbiE8+vmQhUSfuqflnoeP07E75VMQMjCPCL/XZCAW5+SjAvJiamQUB1g7evJjMC2U2eeLL/dhzzaUZvJ5BIGbAytOwX8ck89vEFljd843KBzrOk8= ceph-8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:12:25.834 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T04:12:25.834 INFO:teuthology.orchestra.run.smithi022.stderr:Adding key to root@localhost authorized_keys... 2024-03-24T04:12:25.834 INFO:teuthology.orchestra.run.smithi022.stderr:Adding host smithi022... 2024-03-24T04:12:26.478 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:26 smithi022 ceph-mon[24756]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:27.503 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Added host 'smithi022' with addr '172.21.15.22' 2024-03-24T04:12:28.090 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying mon service with default placement... 2024-03-24T04:12:28.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:28 smithi022 ceph-mon[24756]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "addr": "172.21.15.22", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:28.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:28 smithi022 ceph-mon[24756]: Deploying cephadm binary to smithi022 2024-03-24T04:12:28.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:28 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:28.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:28 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-24T04:12:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:29 smithi022 ceph-mon[24756]: Added host smithi022 2024-03-24T04:12:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:29 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:29.432 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying mgr service with default placement... 2024-03-24T04:12:30.063 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-24T04:12:30.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:30 smithi022 ceph-mon[24756]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:30.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:30 smithi022 ceph-mon[24756]: Saving service mon spec with placement count:5 2024-03-24T04:12:30.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:30 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:30.731 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying crash service with default placement... 2024-03-24T04:12:31.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:31 smithi022 ceph-mon[24756]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:31.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:31 smithi022 ceph-mon[24756]: Saving service mgr spec with placement count:2 2024-03-24T04:12:32.132 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled crash update... 2024-03-24T04:12:32.692 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying prometheus service with default placement... 2024-03-24T04:12:32.969 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:32 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:32.969 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:32 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:32.969 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:32 smithi022 ceph-mon[24756]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:32.969 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:32 smithi022 ceph-mon[24756]: Saving service crash spec with placement * 2024-03-24T04:12:32.970 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:32 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:33.278 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-03-24T04:12:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying grafana service with default placement... 2024-03-24T04:12:34.358 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:34 smithi022 ceph-mon[24756]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:34.358 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:34 smithi022 ceph-mon[24756]: Saving service prometheus spec with placement count:1 2024-03-24T04:12:34.359 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:34 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:34.575 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-03-24T04:12:35.265 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying node-exporter service with default placement... 2024-03-24T04:12:35.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:35 smithi022 ceph-mon[24756]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:35.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:35 smithi022 ceph-mon[24756]: Saving service grafana spec with placement count:1 2024-03-24T04:12:35.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:35 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:35.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:35 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-03-24T04:12:36.581 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying alertmanager service with default placement... 2024-03-24T04:12:36.881 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:36 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:36.881 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:36 smithi022 ceph-mon[24756]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:36.881 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:36 smithi022 ceph-mon[24756]: Saving service node-exporter spec with placement * 2024-03-24T04:12:36.882 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:36 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:37.189 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-03-24T04:12:38.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:38 smithi022 ceph-mon[24756]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:38.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:38 smithi022 ceph-mon[24756]: Saving service alertmanager spec with placement count:1 2024-03-24T04:12:38.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:38 smithi022 ceph-mon[24756]: from='mgr.14118 172.21.15.22:0/910622238' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:40.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:40 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/796789221' entity='client.admin' 2024-03-24T04:12:40.494 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:40 smithi022 ceph-mon[24756]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:12:41.123 INFO:teuthology.orchestra.run.smithi022.stderr:Enabling the dashboard module... 2024-03-24T04:12:42.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:41 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4023920833' entity='client.admin' 2024-03-24T04:12:42.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:41 smithi022 ceph-mon[24756]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:12:42.936 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:42 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1566384067' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T04:12:43.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:43 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1566384067' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T04:12:43.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:43 smithi022 ceph-mon[24756]: mgrmap e9: smithi022.ijpjvk(active, since 23s) 2024-03-24T04:12:43.969 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-24T04:12:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2024-03-24T04:12:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "active_name": "smithi022.ijpjvk", 2024-03-24T04:12:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T04:12:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:44.554 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for the mgr to restart... 2024-03-24T04:12:44.554 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr epoch 9... 2024-03-24T04:12:44.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:44 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1671426191' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T04:12:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2024-03-24T04:12:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-24T04:12:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T04:12:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: mgrmap e10: smithi022.ijpjvk(active, starting, since 0.0527829s) 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:12:49.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:12:49.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:49 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:mgr epoch 9 is available 2024-03-24T04:12:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:Generating a dashboard self-signed certificate... 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:49] ENGINE Bus STARTING 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:49] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: [24/Mar/2024:04:12:49] ENGINE Bus STARTED 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: mgrmap e11: smithi022.ijpjvk(active, since 1.0549s) 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T04:12:50.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:50 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:51.092 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-24T04:12:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial admin user... 2024-03-24T04:12:51.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:51 smithi022 ceph-mon[24756]: mgrmap e12: smithi022.ijpjvk(active, since 2s) 2024-03-24T04:12:51.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:51 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:51.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:51 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:51.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:51 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$gruh7T0sybApZhGUoFNWEugn/b7SzefVdXhuK5yPhokfiwhLzTwZO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711253572, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T04:12:53.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:52 smithi022 ceph-mon[24756]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:53.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:52 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:12:53.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:53 smithi022 ceph-mon[24756]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:12:53.875 INFO:teuthology.orchestra.run.smithi022.stderr:Fetching dashboard port number... 2024-03-24T04:12:54.604 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 8443 2024-03-24T04:12:54.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:54 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2305782023' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T04:12:55.158 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2024-03-24T04:12:55.165 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2024-03-24T04:12:55.166 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2024-03-24T04:12:55.166 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr:Ceph Dashboard is now available at: 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr: URL: https://smithi022.front.sepia.ceph.com:8443/ 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr: User: admin 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr: Password: t7o697pw9h 2024-03-24T04:12:55.168 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:55.169 INFO:teuthology.orchestra.run.smithi022.stderr:Enabling autotune for osd_memory_target 2024-03-24T04:12:55.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:55 smithi022 ceph-mon[24756]: mgrmap e13: smithi022.ijpjvk(active, since 6s) 2024-03-24T04:12:57.135 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr:You can access the Ceph CLI with: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: ceph telemetry on 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr:For more information see: 2024-03-24T04:12:57.680 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.681 INFO:teuthology.orchestra.run.smithi022.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-24T04:12:57.681 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:12:57.681 INFO:teuthology.orchestra.run.smithi022.stderr:Bootstrap complete. 2024-03-24T04:12:57.718 INFO:tasks.cephadm:Fetching config... 2024-03-24T04:12:57.718 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:12:57.718 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T04:12:57.735 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T04:12:57.735 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:12:57.735 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T04:12:57.791 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T04:12:57.791 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:12:57.791 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/keyring of=/dev/stdout 2024-03-24T04:12:57.860 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T04:12:57.860 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:12:57.861 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T04:12:57.915 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T04:12:57.916 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWp7qvVef4Woo8fLPAibFjp8T2lKRtQtRw7sRcce3NquiF0dNizvtiaLo9JZnMkOKCxsR7uqa/aGTiUguxWcAWfWpFxEnT9Azin8o3qtF0BjhnWiBlvzaF5Z3uvRfupANwQSeo2f1CM4CoccUomdf1X4ksJfNYUCJ8/hbVAwpvj/cW1uof9+AdkwuT7r+N2ExfBmNpr0vV1iUiotUY4HHZTBPsrDmqXhN6WGVtBIUEslfnRlOOyxKgAGX7tgs88Qji5MvPFw9Gf/p6sC4ttT0tqcn+FgD1FUwdc/al5mf3MnLUkA2h3vDX2uA3wlMpQJtd1BHCEHGj7lCJ7cop1OxLX9ugZVZOx1TvT6jkhzH1xc2qlPZ5mAG0PwZkBq8K+vEuSpUquMtGiEnsMEKbiE8+vmQhUSfuqflnoeP07E75VMQMjCPCL/XZCAW5+SjAvJiamQUB1g7evJjMC2U2eeLL/dhzzaUZvJ5BIGbAytOwX8ck89vEFljd843KBzrOk8= ceph-8f0be64c-e994-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T04:12:57.997 INFO:teuthology.orchestra.run.smithi022.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWp7qvVef4Woo8fLPAibFjp8T2lKRtQtRw7sRcce3NquiF0dNizvtiaLo9JZnMkOKCxsR7uqa/aGTiUguxWcAWfWpFxEnT9Azin8o3qtF0BjhnWiBlvzaF5Z3uvRfupANwQSeo2f1CM4CoccUomdf1X4ksJfNYUCJ8/hbVAwpvj/cW1uof9+AdkwuT7r+N2ExfBmNpr0vV1iUiotUY4HHZTBPsrDmqXhN6WGVtBIUEslfnRlOOyxKgAGX7tgs88Qji5MvPFw9Gf/p6sC4ttT0tqcn+FgD1FUwdc/al5mf3MnLUkA2h3vDX2uA3wlMpQJtd1BHCEHGj7lCJ7cop1OxLX9ugZVZOx1TvT6jkhzH1xc2qlPZ5mAG0PwZkBq8K+vEuSpUquMtGiEnsMEKbiE8+vmQhUSfuqflnoeP07E75VMQMjCPCL/XZCAW5+SjAvJiamQUB1g7evJjMC2U2eeLL/dhzzaUZvJ5BIGbAytOwX8ck89vEFljd843KBzrOk8= ceph-8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:12:58.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWp7qvVef4Woo8fLPAibFjp8T2lKRtQtRw7sRcce3NquiF0dNizvtiaLo9JZnMkOKCxsR7uqa/aGTiUguxWcAWfWpFxEnT9Azin8o3qtF0BjhnWiBlvzaF5Z3uvRfupANwQSeo2f1CM4CoccUomdf1X4ksJfNYUCJ8/hbVAwpvj/cW1uof9+AdkwuT7r+N2ExfBmNpr0vV1iUiotUY4HHZTBPsrDmqXhN6WGVtBIUEslfnRlOOyxKgAGX7tgs88Qji5MvPFw9Gf/p6sC4ttT0tqcn+FgD1FUwdc/al5mf3MnLUkA2h3vDX2uA3wlMpQJtd1BHCEHGj7lCJ7cop1OxLX9ugZVZOx1TvT6jkhzH1xc2qlPZ5mAG0PwZkBq8K+vEuSpUquMtGiEnsMEKbiE8+vmQhUSfuqflnoeP07E75VMQMjCPCL/XZCAW5+SjAvJiamQUB1g7evJjMC2U2eeLL/dhzzaUZvJ5BIGbAytOwX8ck89vEFljd843KBzrOk8= ceph-8f0be64c-e994-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T04:12:58.057 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWp7qvVef4Woo8fLPAibFjp8T2lKRtQtRw7sRcce3NquiF0dNizvtiaLo9JZnMkOKCxsR7uqa/aGTiUguxWcAWfWpFxEnT9Azin8o3qtF0BjhnWiBlvzaF5Z3uvRfupANwQSeo2f1CM4CoccUomdf1X4ksJfNYUCJ8/hbVAwpvj/cW1uof9+AdkwuT7r+N2ExfBmNpr0vV1iUiotUY4HHZTBPsrDmqXhN6WGVtBIUEslfnRlOOyxKgAGX7tgs88Qji5MvPFw9Gf/p6sC4ttT0tqcn+FgD1FUwdc/al5mf3MnLUkA2h3vDX2uA3wlMpQJtd1BHCEHGj7lCJ7cop1OxLX9ugZVZOx1TvT6jkhzH1xc2qlPZ5mAG0PwZkBq8K+vEuSpUquMtGiEnsMEKbiE8+vmQhUSfuqflnoeP07E75VMQMjCPCL/XZCAW5+SjAvJiamQUB1g7evJjMC2U2eeLL/dhzzaUZvJ5BIGbAytOwX8ck89vEFljd843KBzrOk8= ceph-8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:12:58.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T04:12:58.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:12:58 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3968402660' entity='client.admin' 2024-03-24T04:13:02.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:02 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3208866599' entity='client.admin' 2024-03-24T04:13:02.264 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:02 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:02.264 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:02 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:13:02.264 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:02 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:02.286 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T04:13:02.286 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T04:13:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:03 smithi022 ceph-mon[24756]: Deploying daemon alertmanager.smithi022 on smithi022 2024-03-24T04:13:05.729 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-03-24T04:13:05.729 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:13:05.729 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-03-24T04:13:05.748 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:13:05.748 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:13:05.806 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-03-24T04:13:05.806 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch host add smithi116 2024-03-24T04:13:06.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:05 smithi022 ceph-mon[24756]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:13:06.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:05 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:08 smithi022 ceph-mon[24756]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:10.850 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:10 smithi022 ceph-mon[24756]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:11.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:11 smithi022 ceph-mon[24756]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:13:11.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:11 smithi022 ceph-mon[24756]: Deploying cephadm binary to smithi116 2024-03-24T04:13:12.201 INFO:teuthology.orchestra.run.smithi022.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: Deploying daemon crash.smithi022 on smithi022 2024-03-24T04:13:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:13.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: Added host smithi116 2024-03-24T04:13:13.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:12 smithi022 ceph-mon[24756]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:13.315 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T04:13:14.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:14 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:14.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:14 smithi022 ceph-mon[24756]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:13:15.479 INFO:teuthology.orchestra.run.smithi022.stdout:[{"addr": "172.21.15.22", "hostname": "smithi022", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-03-24T04:13:15.769 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:15 smithi022 ceph-mon[24756]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:13:16.027 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T04:13:16.028 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: Deploying daemon grafana.smithi022 on smithi022 2024-03-24T04:13:17.675 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:17 smithi022 ceph-mon[24756]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:18.489 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-24T04:13:18.792 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:18 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2331223833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T04:13:19.069 INFO:tasks.cephadm:Adding mon.smithi022 on smithi022 2024-03-24T04:13:19.070 INFO:tasks.cephadm:Adding mon.smithi116 on smithi116 2024-03-24T04:13:19.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116' 2024-03-24T04:13:19.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:19 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2331223833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T04:13:19.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:19 smithi022 ceph-mon[24756]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T04:13:19.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:19 smithi022 ceph-mon[24756]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:19.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:19 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:20.657 INFO:teuthology.orchestra.run.smithi116.stdout:Scheduled mon update... 2024-03-24T04:13:21.169 DEBUG:teuthology.orchestra.run.smithi116:mon.smithi116> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi116.service 2024-03-24T04:13:21.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:21.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:22.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:21 smithi022 ceph-mon[24756]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:22.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:21 smithi022 ceph-mon[24756]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:13:22.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:21 smithi022 ceph-mon[24756]: Saving service mon spec with placement smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116;count:2 2024-03-24T04:13:22.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:21 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:23.046 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:23.047 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:23 smithi022 ceph-mon[24756]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:23 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3497430149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:24.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:24.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:25.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:24 smithi022 ceph-mon[24756]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:26.417 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:26.418 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:26.420 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:26.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:26 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/967251871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:27.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:27 smithi022 ceph-mon[24756]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:27.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:27.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:29.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:28 smithi022 ceph-mon[24756]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:29.714 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:29.714 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:29.716 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:30.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:29 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/117849063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:30.795 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:30 smithi022 ceph-mon[24756]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:31.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:31.320 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:32.682 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:32 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:32.682 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:32 smithi022 ceph-mon[24756]: Deploying daemon node-exporter.smithi022 on smithi022 2024-03-24T04:13:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:33.161 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:33.162 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:33.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:33 smithi022 ceph-mon[24756]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:33.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:33 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3133595373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:34.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:34.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:35.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:34 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:35.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:34 smithi022 ceph-mon[24756]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:36.501 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:36.502 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:37 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:37 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T04:13:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:37 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/132422477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:37 smithi022 ceph-mon[24756]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:13:38.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:38.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:38.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:38 smithi022 ceph-mon[24756]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T04:13:38.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:38 smithi022 ceph-mon[24756]: mgrmap e14: smithi022.ijpjvk(active, since 48s) 2024-03-24T04:13:39.971 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:39.971 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:40.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:40 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1041577408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:41.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:41.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:43.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:43 smithi022 ceph-mon[24756]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:13:43.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:43 smithi022 ceph-mon[24756]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:13:43.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:43 smithi022 ceph-mon[24756]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T04:13:43.372 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:43.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:44.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: mgrmap e15: smithi022.ijpjvk(active, starting, since 0.189142s) 2024-03-24T04:13:44.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:13:44.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:44.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:13:44.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:13:44.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/2178598553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:44.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:44.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:44.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:45.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: [24/Mar/2024:04:13:44] ENGINE Bus STARTING 2024-03-24T04:13:45.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: mgrmap e16: smithi022.ijpjvk(active, since 1.19407s) 2024-03-24T04:13:45.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: [24/Mar/2024:04:13:44] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:13:45.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: [24/Mar/2024:04:13:44] ENGINE Bus STARTED 2024-03-24T04:13:45.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:45.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:45.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:46.674 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:46 smithi022 ceph-mon[24756]: mgrmap e17: smithi022.ijpjvk(active, since 2s) 2024-03-24T04:13:47.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:47.395 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:48.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:47 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:48.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:47 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3692537478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:48.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:47 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:48.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:47 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:13:49.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:49.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:49 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:13:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:49 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:13:49.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:13:51.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:51 smithi022 ceph-mon[24756]: Deploying daemon crash.smithi116 on smithi116 2024-03-24T04:13:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:51.490 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:51.492 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:52.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:52 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1051436544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:53.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:53.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:55.043 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:55.045 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:55.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:55 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/76644299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:13:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:56 smithi022 ceph-mon[24756]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:13:56.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:13:56.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:13:58.953 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:13:58.953 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:13:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: Deploying daemon mon.smithi116 on smithi116 2024-03-24T04:13:59.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:13:59 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3898755010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:00.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:14:00.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:14:03.436 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 podman[27132]: 2024-03-24 04:14:03.324029547 +0000 UTC m=+0.121424121 container create e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, name=centos-stream, architecture=x86_64, version=8, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-type=git) 2024-03-24T04:14:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:14:03.447 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:11:45.280671Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T04:14:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 1 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 podman[27132]: 2024-03-24 04:14:03.518428185 +0000 UTC m=+0.315822724 container init e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container) 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 podman[27132]: 2024-03-24 04:14:03.522953385 +0000 UTC m=+0.320347922 container start e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_BRANCH=HEAD, release=754, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream) 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pidfile_write: ignore empty --pid-file 2024-03-24T04:14:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: load: jerasure load: lrc 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: RocksDB version: 6.15.5 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Compile date Apr 18 2022 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: DB SUMMARY 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: DB Session ID: G2YRHEGU89OZR2WW5UGQ 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: CURRENT file: CURRENT 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-24T04:14:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi116/store.db dir, Total Num: 0, files: 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi116/store.db: 000004.log size: 511 ; 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.error_if_exists: 0 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.create_if_missing: 0 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.paranoid_checks: 1 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.env: 0x55ce7f8c6860 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.fs: Posix File System 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.info_log: 0x55ce8171dfa0 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T04:14:03.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.statistics: (nil) 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.use_fsync: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_log_file_size: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_fallocate: 1 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.use_direct_reads: 0 2024-03-24T04:14:03.727 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.db_log_dir: 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi116/store.db 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T04:14:03.728 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.write_buffer_manager: 0x55ce8180e330 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-24T04:14:03.730 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.unordered_write: 0 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T04:14:03.731 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.row_cache: None 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.wal_filter: None 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.preserve_deletes: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.two_write_queues: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.atomic_flush: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T04:14:03.732 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.log_readahead_size: 0 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_background_jobs: 2 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_background_compactions: -1 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_subcompactions: 1 2024-03-24T04:14:03.733 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_open_files: -1 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T04:14:03.734 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_background_flushes: -1 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Compression algorithms supported: 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kZSTD supported: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kXpressCompression supported: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kLZ4Compression supported: 1 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kBZip2Compression supported: 0 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kZlibCompression supported: 1 2024-03-24T04:14:03.735 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: kSnappyCompression supported: 1 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000003 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.merge_operator: 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_filter: None 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T04:14:03.736 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ce816e9d00) 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_type: 0 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_index_type: 0 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_shortening: 1 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: hash_index_allow_collision: 1 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: checksum: 1 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: no_block_cache: 0 2024-03-24T04:14:03.737 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache: 0x55ce81754170 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_options: 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: capacity : 536870912 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_shard_bits : 4 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: strict_capacity_limit : 0 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_compressed: (nil) 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: persistent_cache: (nil) 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size: 4096 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size_deviation: 10 2024-03-24T04:14:03.738 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_restart_interval: 16 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_block_restart_interval: 1 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_block_size: 4096 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: partition_filters: 0 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: use_delta_encoding: 1 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: whole_key_filtering: 1 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: verify_compression: 0 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: format_version: 4 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_index_compression: 1 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_align: 0 2024-03-24T04:14:03.739 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression: NoCompression 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.num_levels: 7 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T04:14:03.740 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T04:14:03.741 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T04:14:03.742 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.arena_block_size: 4194304 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T04:14:03.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.table_properties_collectors: 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.inplace_update_support: 0 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T04:14:03.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.bloom_locality: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.max_successive_merges: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.ttl: 2592000 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.enable_blob_files: false 2024-03-24T04:14:03.745 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.min_blob_size: 0 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253643548999, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-24T04:14:03.746 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253643549414, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711253643, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "98eb171d-6008-4e52-be89-4ff387b0525a", "db_session_id": "G2YRHEGU89OZR2WW5UGQ"}} 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711253643549824, "job": 1, "event": "recovery_finished"} 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi116/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55ce8173aa80 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: DB pointer 0x55ce8174a000 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** DB Stats ** 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:14:03.747 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.748 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:14:03.749 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.750 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:14:03.751 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116 does not exist in monmap, will attempt to join an existing cluster 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: using public_addr v2:172.21.15.116:0/0 -> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: starting mon.smithi116 rank -1 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi116 fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(???) e0 preinit fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).mds e1 new map 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).mds e1 print_map 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: e1 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:14:03.752 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: legacy client fscid: -1 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: No filesystems configured 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:14:03.753 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon crash.smithi022 on smithi022 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Added host smithi116 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.754 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon grafana.smithi022 on smithi022 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2331223833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2331223833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T04:14:03.755 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Saving service mon spec with placement smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116;count:2 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3497430149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/967251871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/117849063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon node-exporter.smithi022 on smithi022 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3133595373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.757 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/132422477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14160 172.21.15.22:0/774381434' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mgrmap e14: smithi022.ijpjvk(active, since 48s) 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1041577408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:14:03.758 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mgrmap e15: smithi022.ijpjvk(active, starting, since 0.189142s) 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.759 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/2178598553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: [24/Mar/2024:04:13:44] ENGINE Bus STARTING 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mgrmap e16: smithi022.ijpjvk(active, since 1.19407s) 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: [24/Mar/2024:04:13:44] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: [24/Mar/2024:04:13:44] ENGINE Bus STARTED 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mgrmap e17: smithi022.ijpjvk(active, since 2s) 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.760 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3692537478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T04:14:03.761 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon crash.smithi116 on smithi116 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1051436544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/76644299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:03.762 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: Deploying daemon mon.smithi116 on smithi116 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3898755010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 ceph-mon[27165]: mon.smithi116@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 bash[27132]: e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 2024-03-24T04:14:03.763 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:03 smithi116 systemd[1]: Started Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:14:05.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T04:14:05.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T04:14:08.912 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:08.912 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: mon.smithi022 calling monitor election 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: mon.smithi116 calling monitor election 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: fsmap 2024-03-24T04:14:08.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: mgrmap e17: smithi022.ijpjvk(active, since 25s) 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: overall HEALTH_OK 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:08.914 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:08 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:14:09.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: mon.smithi022 calling monitor election 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: mon.smithi116 calling monitor election 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:14:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: fsmap 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: mgrmap e17: smithi022.ijpjvk(active, since 25s) 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: overall HEALTH_OK 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:09.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:08 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3725934759' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:14:09.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:09 smithi116 ceph-mon[27165]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-24T04:14:09.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:09 smithi116 ceph-mon[27165]: mgrmap e18: smithi022.ijpjvk(active, since 25s), standbys: smithi116.oxktjt 2024-03-24T04:14:09.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:09 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:14:09.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:09 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:10.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:09 smithi022 ceph-mon[24756]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-24T04:14:10.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:09 smithi022 ceph-mon[24756]: mgrmap e18: smithi022.ijpjvk(active, since 25s), standbys: smithi116.oxktjt 2024-03-24T04:14:10.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:09 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:14:10.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:09 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:14:10.093 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T04:14:10.093 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","modified":"2024-03-24T04:14:03.574607Z","created":"2024-03-24T04:11:45.280671Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi116","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T04:14:10.094 INFO:teuthology.orchestra.run.smithi116.stderr:dumped monmap epoch 2 2024-03-24T04:14:10.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:10 smithi022 ceph-mon[24756]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:10.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:10 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/2122482877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:10.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:10 smithi116 ceph-mon[27165]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:10.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:10 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/2122482877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T04:14:11.376 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T04:14:11.376 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T04:14:12.877 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:12 smithi022 ceph-mon[24756]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:12.877 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:12 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:12 smithi116 ceph-mon[27165]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:12 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:13.302 INFO:teuthology.orchestra.run.smithi022.stdout:# minimal ceph.conf for 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:14:13.302 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-24T04:14:13.302 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:14:13.302 INFO:teuthology.orchestra.run.smithi022.stdout: mon_host = [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-03-24T04:14:13.859 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:13 smithi022 ceph-mon[24756]: Deploying daemon prometheus.smithi022 on smithi022 2024-03-24T04:14:13.859 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:13 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:13.859 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:13 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1743558442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:13.860 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T04:14:13.861 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:14:13.861 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T04:14:13.890 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:14:13.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:14:13.958 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:14:13.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T04:14:13.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:13 smithi116 ceph-mon[27165]: Deploying daemon prometheus.smithi022 on smithi022 2024-03-24T04:14:13.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:13 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:13.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:13 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1743558442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:13.993 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:14:13.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:14:14.063 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T04:14:14.063 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:14:14.063 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T04:14:14.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T04:14:14.078 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-24T04:14:14.132 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T04:14:14.132 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 04:13:32.359589163 +0000 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 04:10:54.421871554 +0000 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 04:10:54.421871554 +0000 2024-03-24T04:14:14.133 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 04:10:54.421871554 +0000 2024-03-24T04:14:14.133 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T04:14:14.197 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T04:14:14.197 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T04:14:14.197 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.0728e-05 s, 5.6 MB/s 2024-03-24T04:14:14.198 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T04:14:14.253 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 04:13:32.360589151 +0000 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 04:10:54.749867793 +0000 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 04:10:54.749867793 +0000 2024-03-24T04:14:14.309 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 04:10:54.749867793 +0000 2024-03-24T04:14:14.309 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T04:14:14.373 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T04:14:14.374 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T04:14:14.374 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.6498e-05 s, 5.3 MB/s 2024-03-24T04:14:14.375 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T04:14:14.430 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:14.485 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 04:13:32.361589139 +0000 2024-03-24T04:14:14.486 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 04:10:55.087863917 +0000 2024-03-24T04:14:14.486 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 04:10:55.087863917 +0000 2024-03-24T04:14:14.486 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 04:10:55.087863917 +0000 2024-03-24T04:14:14.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T04:14:14.550 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T04:14:14.550 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T04:14:14.550 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 8.7295e-05 s, 5.9 MB/s 2024-03-24T04:14:14.551 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T04:14:14.607 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 04:13:32.361589139 +0000 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 04:10:55.411860202 +0000 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 04:10:55.411860202 +0000 2024-03-24T04:14:14.662 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 04:10:55.411860202 +0000 2024-03-24T04:14:14.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T04:14:14.728 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T04:14:14.728 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T04:14:14.728 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 8.3737e-05 s, 6.1 MB/s 2024-03-24T04:14:14.729 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T04:14:14.786 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T04:14:14.786 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T04:14:14.805 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T04:14:14.805 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-24T04:14:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T04:14:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:14.863 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T04:14:14.863 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:14.863 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:14.864 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 04:14:09.206698834 +0000 2024-03-24T04:14:14.864 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 04:10:48.075102674 +0000 2024-03-24T04:14:14.864 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 04:10:48.075102674 +0000 2024-03-24T04:14:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 04:10:48.075102674 +0000 2024-03-24T04:14:14.864 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T04:14:14.930 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:14 smithi116 ceph-mon[27165]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T04:14:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T04:14:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000236121 s, 2.2 MB/s 2024-03-24T04:14:14.934 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T04:14:14.992 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-24T04:14:15.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:14 smithi022 ceph-mon[24756]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 04:14:09.207698818 +0000 2024-03-24T04:14:15.049 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 04:10:48.399097433 +0000 2024-03-24T04:14:15.050 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 04:10:48.399097433 +0000 2024-03-24T04:14:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 04:10:48.399097433 +0000 2024-03-24T04:14:15.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T04:14:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T04:14:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T04:14:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000181897 s, 2.8 MB/s 2024-03-24T04:14:15.119 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T04:14:15.176 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 04:14:09.207698818 +0000 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 04:10:48.734092014 +0000 2024-03-24T04:14:15.232 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 04:10:48.734092014 +0000 2024-03-24T04:14:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 04:10:48.734092014 +0000 2024-03-24T04:14:15.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T04:14:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T04:14:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T04:14:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000198752 s, 2.6 MB/s 2024-03-24T04:14:15.299 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T04:14:15.355 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 04:14:09.208698802 +0000 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 04:10:49.074086513 +0000 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 04:10:49.074086513 +0000 2024-03-24T04:14:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 04:10:49.074086513 +0000 2024-03-24T04:14:15.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T04:14:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T04:14:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T04:14:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000246704 s, 2.1 MB/s 2024-03-24T04:14:15.477 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T04:14:15.532 INFO:tasks.cephadm:Deploying osd.0 on smithi022 with /dev/vg_nvme/lv_4... 2024-03-24T04:14:15.533 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T04:14:16.921 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:16 smithi022 ceph-mon[24756]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:16.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:16 smithi116 ceph-mon[27165]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:18.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:18 smithi022 ceph-mon[24756]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:18 smithi116 ceph-mon[27165]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00850919 s, 1.2 GB/s 2024-03-24T04:14:20.436 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T04:14:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:20 smithi116 ceph-mon[27165]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:20.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:20 smithi022 ceph-mon[24756]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:21.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_4 2024-03-24T04:14:22.539 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:22 smithi022 ceph-mon[24756]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:22 smithi116 ceph-mon[27165]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:24.682 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:24 smithi022 ceph-mon[24756]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:24 smithi116 ceph-mon[27165]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:14:26.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:14:26.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:14:26.647 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:28.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:28.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:27 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:14:28.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:27 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:14:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:29.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:14:29.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4036310245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f12be2f-9abd-44d0-9be5-d478e00bf4d9"}]: dispatch 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4036310245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f12be2f-9abd-44d0-9be5-d478e00bf4d9"}]': finished 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:29.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:28 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1037228855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:14:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:14:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4036310245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f12be2f-9abd-44d0-9be5-d478e00bf4d9"}]: dispatch 2024-03-24T04:14:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4036310245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f12be2f-9abd-44d0-9be5-d478e00bf4d9"}]': finished 2024-03-24T04:14:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T04:14:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:28 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1037228855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:14:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:30 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:31.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:31.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:31.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:14:31.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:14:31.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:30 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:32.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:31 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:14:32.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:31 smithi116 ceph-mon[27165]: Reconfiguring mon.smithi022 (unknown last config time)... 2024-03-24T04:14:32.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:31 smithi116 ceph-mon[27165]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T04:14:32.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:31 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:14:32.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:31 smithi022 ceph-mon[24756]: Reconfiguring mon.smithi022 (unknown last config time)... 2024-03-24T04:14:32.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:31 smithi022 ceph-mon[24756]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T04:14:33.040 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:32 smithi022 ceph-mon[24756]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:33.040 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:33.041 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:14:33.041 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:14:33.041 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:33.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:32 smithi116 ceph-mon[27165]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:33.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:33.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:14:33.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:14:33.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:34.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:33 smithi116 ceph-mon[27165]: Reconfiguring mgr.smithi022.ijpjvk (unknown last config time)... 2024-03-24T04:14:34.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:33 smithi116 ceph-mon[27165]: Reconfiguring daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:14:34.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:33 smithi022 ceph-mon[24756]: Reconfiguring mgr.smithi022.ijpjvk (unknown last config time)... 2024-03-24T04:14:34.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:33 smithi022 ceph-mon[24756]: Reconfiguring daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:14:35.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:34 smithi022 ceph-mon[24756]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:35.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:34 smithi116 ceph-mon[27165]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:36 smithi116 ceph-mon[27165]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T04:14:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:36 smithi116 ceph-mon[27165]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T04:14:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:36 smithi116 ceph-mon[27165]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:36.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:36.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:36 smithi022 ceph-mon[24756]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T04:14:36.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:36 smithi022 ceph-mon[24756]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T04:14:36.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:36 smithi022 ceph-mon[24756]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:14:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:37 smithi022 ceph-mon[24756]: Deploying daemon osd.0 on smithi022 2024-03-24T04:14:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:14:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:37 smithi116 ceph-mon[27165]: Deploying daemon osd.0 on smithi022 2024-03-24T04:14:38.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:38 smithi022 ceph-mon[24756]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:38 smithi116 ceph-mon[27165]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:40.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:40 smithi022 ceph-mon[24756]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:40 smithi116 ceph-mon[27165]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:41.904 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:41.904 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:41 smithi022 ceph-mon[24756]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T04:14:41.904 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:41.904 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:41.905 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:41 smithi022 ceph-mon[24756]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T04:14:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:41 smithi116 ceph-mon[27165]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T04:14:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:42.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:42.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:41 smithi116 ceph-mon[27165]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T04:14:42.903 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:42 smithi022 ceph-mon[24756]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:43.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:42 smithi116 ceph-mon[27165]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:43.928 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:14:43.928 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:14:44.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:14:44.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:14:45.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:44 smithi022 ceph-mon[24756]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:44 smithi116 ceph-mon[27165]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:46.923 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:46 smithi022 ceph-mon[24756]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:46.923 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:46 smithi116 ceph-mon[27165]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:49 smithi116 ceph-mon[27165]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:49 smithi116 ceph-mon[27165]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T04:14:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:49 smithi116 ceph-mon[27165]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T04:14:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:49 smithi116 ceph-mon[27165]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T04:14:49.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:49 smithi022 ceph-mon[24756]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:49.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:49.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:49 smithi022 ceph-mon[24756]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T04:14:49.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:49 smithi022 ceph-mon[24756]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T04:14:49.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:49 smithi022 ceph-mon[24756]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T04:14:49.578 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 0 on host 'smithi022' 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:14:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:50 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T04:14:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T04:14:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:14:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:50 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:50.807 DEBUG:teuthology.orchestra.run.smithi022:osd.0> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T04:14:50.809 INFO:tasks.cephadm:Deploying osd.1 on smithi022 with /dev/vg_nvme/lv_3... 2024-03-24T04:14:50.810 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T04:14:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:51 smithi022 ceph-mon[24756]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:14:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:51 smithi022 ceph-mon[24756]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T04:14:51.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:51.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:51.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:51 smithi116 ceph-mon[27165]: from='osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:14:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:51 smithi116 ceph-mon[27165]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T04:14:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:51 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:51 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:51 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723] boot 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T04:14:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:52 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:14:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:14:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T04:14:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: osd.0 [v2:172.21.15.22:6802/1971225723,v1:172.21.15.22:6803/1971225723] boot 2024-03-24T04:14:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T04:14:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:52 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:14:53.546 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:53 smithi022 ceph-mon[24756]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T04:14:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:53 smithi116 ceph-mon[27165]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T04:14:54.535 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:54 smithi022 ceph-mon[24756]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:54 smithi116 ceph-mon[27165]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00811022 s, 1.3 GB/s 2024-03-24T04:14:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T04:14:56.538 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:56 smithi022 ceph-mon[24756]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:56.538 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:56 smithi116 ceph-mon[27165]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:57.075 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_3 2024-03-24T04:14:57.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:57 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:57.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:57 smithi022 ceph-mon[24756]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-24T04:14:57.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:57 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:57.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:57 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:57.687 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:57 smithi022 ceph-mon[24756]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-24T04:14:57.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:57 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:57.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:57 smithi116 ceph-mon[27165]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-24T04:14:57.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:57 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:14:57.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:57 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:57.782 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:57 smithi116 ceph-mon[27165]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-24T04:14:59.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: Reconfiguring mgr.smithi116.oxktjt (monmap changed)... 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:59.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:58 smithi022 ceph-mon[24756]: Reconfiguring daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:14:59.040 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:14:59.040 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:59.040 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: Reconfiguring mgr.smithi116.oxktjt (monmap changed)... 2024-03-24T04:14:59.041 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:14:59.041 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:14:59.041 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:59.041 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:58 smithi116 ceph-mon[27165]: Reconfiguring daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-24T04:14:59.917 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:14:59.918 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:14:59.918 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:14:59.918 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:14:59 smithi116 ceph-mon[27165]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-24T04:15:00.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:15:00.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:15:00.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:00.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:14:59 smithi022 ceph-mon[24756]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-24T04:15:01.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:01 smithi116 ceph-mon[27165]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:01 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:01 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:15:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:01 smithi022 ceph-mon[24756]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:01 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:01 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:15:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:15:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/382478652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5da2b7-2441-4bec-98be-21c5dd13f2c2"}]: dispatch 2024-03-24T04:15:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/382478652' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5da2b7-2441-4bec-98be-21c5dd13f2c2"}]': finished 2024-03-24T04:15:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T04:15:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:02 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2304448300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:02.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:15:02.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/382478652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5da2b7-2441-4bec-98be-21c5dd13f2c2"}]: dispatch 2024-03-24T04:15:02.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/382478652' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5da2b7-2441-4bec-98be-21c5dd13f2c2"}]': finished 2024-03-24T04:15:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T04:15:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:02 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2304448300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:03.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:03 smithi116 ceph-mon[27165]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:03 smithi022 ceph-mon[24756]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:05.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:05 smithi022 ceph-mon[24756]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:05 smithi116 ceph-mon[27165]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:07.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:07 smithi116 ceph-mon[27165]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:07.480 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:07 smithi022 ceph-mon[24756]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:08.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:08 smithi022 ceph-mon[24756]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:08.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:15:08.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:08.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:08 smithi022 ceph-mon[24756]: Deploying daemon osd.1 on smithi022 2024-03-24T04:15:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:08 smithi116 ceph-mon[27165]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:15:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:08.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:08 smithi116 ceph-mon[27165]: Deploying daemon osd.1 on smithi022 2024-03-24T04:15:10.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:10 smithi022 ceph-mon[24756]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:10 smithi116 ceph-mon[27165]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:12.379 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:12 smithi022 ceph-mon[24756]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:12 smithi116 ceph-mon[27165]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:14.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:13 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:14.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:13 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:15.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:14 smithi022 ceph-mon[24756]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:14 smithi116 ceph-mon[27165]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:15.626 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 1 on host 'smithi022' 2024-03-24T04:15:16.229 DEBUG:teuthology.orchestra.run.smithi022:osd.1> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T04:15:16.232 INFO:tasks.cephadm:Deploying osd.2 on smithi022 with /dev/vg_nvme/lv_2... 2024-03-24T04:15:16.232 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:15:16.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.22:3000"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.22:3000"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.22:9095"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.22:9095"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:16.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:16 smithi116 ceph-mon[27165]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.22:3000"}]: dispatch 2024-03-24T04:15:17.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.22:3000"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.22:9095"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.22:9095"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:17.025 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:16 smithi022 ceph-mon[24756]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T04:15:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-mon[24756]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T04:15:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-mon[24756]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T04:15:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-mon[24756]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:15:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:18.298 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[38016]: 2024-03-24T04:15:18.006+0000 7faa751ec700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T04:15:18.299 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:18 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[38016]: 2024-03-24T04:15:18.012+0000 7faa6e380700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:15:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:18 smithi116 ceph-mon[27165]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T04:15:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:18 smithi116 ceph-mon[27165]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T04:15:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:18 smithi116 ceph-mon[27165]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:15:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:19.115 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T04:15:19.115 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T04:15:19.115 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T04:15:19.115 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T04:15:19.116 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00970186 s, 1.1 GB/s 2024-03-24T04:15:19.116 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T04:15:19.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:19 smithi022 ceph-mon[24756]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:19.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:19 smithi022 ceph-mon[24756]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:15:19.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:19 smithi022 ceph-mon[24756]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T04:15:19.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:19 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:19.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:19 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:19 smithi116 ceph-mon[27165]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:19 smithi116 ceph-mon[27165]: from='osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:15:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:19 smithi116 ceph-mon[27165]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T04:15:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:19 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:19 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:19.754 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_2 2024-03-24T04:15:20.254 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:15:20.254 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:15:20.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:20.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854] boot 2024-03-24T04:15:20.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T04:15:20.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:15:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:15:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: osd.1 [v2:172.21.15.22:6810/2734658854,v1:172.21.15.22:6811/2734658854] boot 2024-03-24T04:15:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T04:15:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:15:21.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:21 smithi022 ceph-mon[24756]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:21.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:21 smithi022 ceph-mon[24756]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T04:15:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:21 smithi116 ceph-mon[27165]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-24T04:15:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:21 smithi116 ceph-mon[27165]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T04:15:23.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:23 smithi022 ceph-mon[24756]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:23.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:15:23.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:15:23.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:23 smithi116 ceph-mon[27165]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:15:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:15:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:24.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:24 smithi022 ceph-mon[24756]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:15:24.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:24.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:24 smithi116 ceph-mon[27165]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:15:24.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:25.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:25 smithi022 ceph-mon[24756]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:25 smithi116 ceph-mon[27165]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:26.123 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:26 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4207858295' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a27c891-0e0b-4c8b-8404-334ff29c44f7"}]: dispatch 2024-03-24T04:15:26.123 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:26 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4207858295' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a27c891-0e0b-4c8b-8404-334ff29c44f7"}]': finished 2024-03-24T04:15:26.124 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:26 smithi022 ceph-mon[24756]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T04:15:26.124 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:26 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:26 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4207858295' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a27c891-0e0b-4c8b-8404-334ff29c44f7"}]: dispatch 2024-03-24T04:15:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:26 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4207858295' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a27c891-0e0b-4c8b-8404-334ff29c44f7"}]': finished 2024-03-24T04:15:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:26 smithi116 ceph-mon[27165]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T04:15:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:26 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:27.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:27 smithi022 ceph-mon[24756]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:27.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:27 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/658920161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:27 smithi116 ceph-mon[27165]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:27 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/658920161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:29.296 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:29 smithi116 ceph-mon[27165]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:29.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:29 smithi022 ceph-mon[24756]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:31.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:32.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:32 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi022 2024-03-24T04:15:32.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:32 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi022 to 6434M 2024-03-24T04:15:32.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:32 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi022 2024-03-24T04:15:32.364 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:32 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi022 to 6434M 2024-03-24T04:15:33.137 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:33 smithi022 ceph-mon[24756]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:33.138 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:33 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:33.138 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:33 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:15:33.138 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:33 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:33 smithi116 ceph-mon[27165]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:33 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:33 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:15:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:33 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:34 smithi022 ceph-mon[24756]: Deploying daemon osd.2 on smithi022 2024-03-24T04:15:34.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:34 smithi116 ceph-mon[27165]: Deploying daemon osd.2 on smithi022 2024-03-24T04:15:35.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:35 smithi022 ceph-mon[24756]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:35 smithi116 ceph-mon[27165]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:37.153 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:37 smithi022 ceph-mon[24756]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:37 smithi116 ceph-mon[27165]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:39.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:39 smithi022 ceph-mon[24756]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:39.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:39 smithi116 ceph-mon[27165]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:41.364 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:41 smithi022 ceph-mon[24756]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:41.364 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:41 smithi116 ceph-mon[27165]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:43.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:43.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:43.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:43.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:43.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:43.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:43.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:15:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:15:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:44 smithi022 ceph-mon[24756]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T04:15:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:15:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:15:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:44 smithi116 ceph-mon[27165]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T04:15:45.078 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 2 on host 'smithi022' 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:45.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:45.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:45 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:45.747 DEBUG:teuthology.orchestra.run.smithi022:osd.2> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T04:15:45.750 INFO:tasks.cephadm:Deploying osd.3 on smithi022 with /dev/vg_nvme/lv_1... 2024-03-24T04:15:45.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T04:15:46.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:46 smithi022 ceph-mon[24756]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:15:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:46 smithi022 ceph-mon[24756]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T04:15:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:46 smithi116 ceph-mon[27165]: from='osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:15:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:46 smithi116 ceph-mon[27165]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T04:15:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:46.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:47.386 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:15:47.386 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:15:47.386 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:47.386 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:47.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546] boot 2024-03-24T04:15:47.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T04:15:47.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:47 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:47.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:15:47.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:15:47.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T04:15:47.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:47.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: osd.2 [v2:172.21.15.22:6818/3565337546,v1:172.21.15.22:6819/3565337546] boot 2024-03-24T04:15:47.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T04:15:47.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:47 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:15:48.422 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:48 smithi022 ceph-mon[24756]: pgmap v69: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:48.423 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:48 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T04:15:48.423 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:48 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T04:15:48.423 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:48 smithi022 ceph-mon[24756]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T04:15:48.423 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:48 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T04:15:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:48 smithi116 ceph-mon[27165]: pgmap v69: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:48 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T04:15:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:48 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T04:15:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:48 smithi116 ceph-mon[27165]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T04:15:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:48 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00998003 s, 1.1 GB/s 2024-03-24T04:15:49.405 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T04:15:50.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:50 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T04:15:50.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:50 smithi022 ceph-mon[24756]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T04:15:50.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:50 smithi022 ceph-mon[24756]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:50.376 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43291]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T04:15:50.376 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43291]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:50.376 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43291]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:50.379 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_1 2024-03-24T04:15:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:50 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T04:15:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:50 smithi116 ceph-mon[27165]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T04:15:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:50 smithi116 ceph-mon[27165]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:50.675 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43298]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T04:15:50.675 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43298]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:50.675 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43298]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:50.939 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43399]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T04:15:50.939 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43399]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:50.939 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43399]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:51.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 ceph-mon[24756]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T04:15:51.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:51.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:51.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43433]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T04:15:51.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43433]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:51.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:15:50 smithi022 sudo[43433]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:51.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43449]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T04:15:51.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43449]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:51.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43449]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:51 smithi116 ceph-mon[27165]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T04:15:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:51 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:51 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:51.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43455]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T04:15:51.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43455]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:51.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43455]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:51.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43458]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T04:15:51.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43458]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:51.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43458]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:52.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43496]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T04:15:52.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43496]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:52.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:51 smithi022 sudo[43496]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:52.489 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29483]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T04:15:52.489 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29483]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:52.489 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29483]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:52.489 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29486]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T04:15:52.490 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29486]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T04:15:52.490 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 sudo[29486]: pam_unix(sudo:session): session closed for user root 2024-03-24T04:15:52.793 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: pgmap v74: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:15:52.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:52 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:15:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: pgmap v74: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T04:15:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T04:15:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:15:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:15:52.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:15:52.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:52 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: mgrmap e19: smithi022.ijpjvk(active, since 2m), standbys: smithi116.oxktjt 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:15:53.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:53 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T04:15:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T04:15:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: mgrmap e19: smithi022.ijpjvk(active, since 2m), standbys: smithi116.oxktjt 2024-03-24T04:15:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:15:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:15:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:15:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:53 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:54 smithi022 ceph-mon[24756]: pgmap v75: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:54 smithi116 ceph-mon[27165]: pgmap v75: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:15:56.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi022 to 4289M 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:56.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:55 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi022 to 4289M 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:15:56.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:55 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:15:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:56 smithi116 ceph-mon[27165]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.3 KiB/s rd, 254 KiB/s wr, 17 op/s 2024-03-24T04:15:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:56 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/88820480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69"}]: dispatch 2024-03-24T04:15:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:56 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/88820480' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69"}]': finished 2024-03-24T04:15:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:56 smithi116 ceph-mon[27165]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T04:15:57.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:56 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:15:57.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:56 smithi022 ceph-mon[24756]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.3 KiB/s rd, 254 KiB/s wr, 17 op/s 2024-03-24T04:15:57.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:56 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/88820480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69"}]: dispatch 2024-03-24T04:15:57.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:56 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/88820480' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69"}]': finished 2024-03-24T04:15:57.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:56 smithi022 ceph-mon[24756]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T04:15:57.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:56 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:15:58.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:57 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2266974656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:58.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:57 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2266974656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:15:59.145 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:15:58 smithi022 ceph-mon[24756]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T04:15:59.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:15:58 smithi116 ceph-mon[27165]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T04:16:01.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:01 smithi022 ceph-mon[24756]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 143 KiB/s wr, 9 op/s 2024-03-24T04:16:01.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:01 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:01.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:01 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:01.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:01 smithi116 ceph-mon[27165]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 143 KiB/s wr, 9 op/s 2024-03-24T04:16:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:01 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:01 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:03.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:03 smithi022 ceph-mon[24756]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T04:16:03.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:16:03.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:03.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:03 smithi116 ceph-mon[27165]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T04:16:03.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:16:03.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:04.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:04 smithi022 ceph-mon[24756]: Deploying daemon osd.3 on smithi022 2024-03-24T04:16:04.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:04 smithi116 ceph-mon[27165]: Deploying daemon osd.3 on smithi022 2024-03-24T04:16:05.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:05 smithi022 ceph-mon[24756]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T04:16:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:05 smithi116 ceph-mon[27165]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-24T04:16:07.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:07 smithi022 ceph-mon[24756]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:07.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:07 smithi116 ceph-mon[27165]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:08.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:08 smithi116 ceph-mon[27165]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:08.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:08.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:08.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:08.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:09.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:08 smithi022 ceph-mon[24756]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:09.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:10.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:10 smithi116 ceph-mon[27165]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:10 smithi022 ceph-mon[24756]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:11.510 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 3 on host 'smithi022' 2024-03-24T04:16:11.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:11 smithi022 ceph-mon[24756]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:16:11.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:11 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:11.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:11 smithi116 ceph-mon[27165]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:16:11.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:11 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:12.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:12 smithi022 ceph-mon[24756]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:12.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:12 smithi022 ceph-mon[24756]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T04:16:12.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:12 smithi022 ceph-mon[24756]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T04:16:12.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:12 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:12.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:12 smithi022 ceph-mon[24756]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:16:12.822 DEBUG:teuthology.orchestra.run.smithi022:osd.3> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T04:16:12.824 INFO:tasks.cephadm:Deploying osd.4 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-24T04:16:12.824 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T04:16:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:12 smithi116 ceph-mon[27165]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:12 smithi116 ceph-mon[27165]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T04:16:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:12 smithi116 ceph-mon[27165]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T04:16:12.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:12 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:12.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:12 smithi116 ceph-mon[27165]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:16:13.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:13 smithi022 ceph-mon[24756]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:16:13.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:13 smithi022 ceph-mon[24756]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T04:16:13.963 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:13 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:13.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:13 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:13.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:13 smithi116 ceph-mon[27165]: from='osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T04:16:13.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:13 smithi116 ceph-mon[27165]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T04:16:13.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:13 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:13.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:13 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:14.774 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T04:16:14.774 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T04:16:14.774 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T04:16:14.774 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T04:16:14.774 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113557 s, 923 MB/s 2024-03-24T04:16:14.775 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T04:16:14.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:16:14.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:16:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823] boot 2024-03-24T04:16:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T04:16:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:14.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:14 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:14.983 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:16:14.983 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:16:14.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T04:16:14.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: osd.3 [v2:172.21.15.22:6826/2272173823,v1:172.21.15.22:6827/2272173823] boot 2024-03-24T04:16:14.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T04:16:14.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:16:14.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:14 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:15.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-24T04:16:15.756 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:15 smithi116 ceph-mon[27165]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T04:16:15.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:15 smithi022 ceph-mon[24756]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T04:16:16.970 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:16 smithi116 ceph-mon[27165]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:17.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:16 smithi022 ceph-mon[24756]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi022 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi022 to 3217M 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:18.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:18 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:18.591 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi022 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi022 to 3217M 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:18.592 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:18.593 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:18 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:20.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/538944090' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]: dispatch 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]: dispatch 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]': finished 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:20.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:20 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1253395497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/538944090' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]: dispatch 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]: dispatch 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1de55b84-e20c-434a-8890-25b0a56e2093"}]': finished 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:20.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:20 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1253395497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:16:22.783 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:22 smithi116 ceph-mon[27165]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:22.783 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:22.783 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:23.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:22 smithi022 ceph-mon[24756]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:23.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:23.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:24.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:24 smithi116 ceph-mon[27165]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:25.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:24 smithi022 ceph-mon[24756]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:25.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:25 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:16:25.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:25 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:26.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:25 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:16:26.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:25 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:26.875 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:26 smithi116 ceph-mon[27165]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:26.875 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:26 smithi116 ceph-mon[27165]: Deploying daemon osd.4 on smithi116 2024-03-24T04:16:27.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:26 smithi022 ceph-mon[24756]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:27.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:26 smithi022 ceph-mon[24756]: Deploying daemon osd.4 on smithi116 2024-03-24T04:16:28.941 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:28 smithi116 ceph-mon[27165]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:29.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:28 smithi022 ceph-mon[24756]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:31 smithi022 ceph-mon[24756]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:31 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:31 smithi116 ceph-mon[27165]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:31 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:33.240 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 4 on host 'smithi116' 2024-03-24T04:16:33.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:33 smithi022 ceph-mon[24756]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:33 smithi116 ceph-mon[27165]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:34.053 DEBUG:teuthology.orchestra.run.smithi116:osd.4> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T04:16:34.056 INFO:tasks.cephadm:Deploying osd.5 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-24T04:16:34.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T04:16:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:34 smithi022 ceph-mon[24756]: from='osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:16:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:34 smithi022 ceph-mon[24756]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:16:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:34 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:34 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:34.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:34 smithi116 ceph-mon[27165]: from='osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:16:34.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:34 smithi116 ceph-mon[27165]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:16:34.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:34 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:34.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:34 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:35.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:35.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T04:16:35.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T04:16:35.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:35.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: from='osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:35.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:35 smithi022 ceph-mon[24756]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: from='osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:35.407 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-mon[27165]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:35.408 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[31350]: 2024-03-24T04:16:35.037+0000 7f742c5b5700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T04:16:35.408 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:16:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[31350]: 2024-03-24T04:16:35.041+0000 7f7426f4c700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:16:36.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:36 smithi116 ceph-mon[27165]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:16:36.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:36 smithi116 ceph-mon[27165]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T04:16:36.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:36.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:36 smithi022 ceph-mon[24756]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:16:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:36 smithi022 ceph-mon[24756]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T04:16:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:37.162 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128687 s, 815 MB/s 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 2024-03-24T04:16:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T04:16:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:16:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:16:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323] boot 2024-03-24T04:16:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T04:16:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:16:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:16:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T04:16:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: osd.4 [v2:172.21.15.116:6800/3256324323,v1:172.21.15.116:6801/3256324323] boot 2024-03-24T04:16:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T04:16:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:16:38.250 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:38 smithi116 ceph-mon[27165]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T04:16:38.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:38 smithi022 ceph-mon[24756]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T04:16:38.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:39.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:39.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:39 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:39 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:40.179 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:40 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi116 2024-03-24T04:16:40.180 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:40 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi116 to 15936M 2024-03-24T04:16:40.180 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:40 smithi116 ceph-mon[27165]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T04:16:40.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:40 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi116 2024-03-24T04:16:40.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:40 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi116 to 15936M 2024-03-24T04:16:40.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:40 smithi022 ceph-mon[24756]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T04:16:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:41 smithi116 ceph-mon[27165]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:41 smithi116 ceph-mon[27165]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T04:16:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:16:41.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:16:41.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:41 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:41.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:41 smithi022 ceph-mon[24756]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:41.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:41 smithi022 ceph-mon[24756]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T04:16:41.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:16:41.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:16:41.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:41 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:42 smithi116 ceph-mon[27165]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:16:42.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:42 smithi022 ceph-mon[24756]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1074523137' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]: dispatch 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]: dispatch 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]': finished 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T04:16:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:43.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-24T04:16:43.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1074523137' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]: dispatch 2024-03-24T04:16:43.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]: dispatch 2024-03-24T04:16:43.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b426fc4-8e82-402b-90ca-dfb3bea82176"}]': finished 2024-03-24T04:16:43.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T04:16:43.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:44 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3434561626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:16:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:16:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:16:44.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:44 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3434561626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:16:44.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:16:44.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:16:45.318 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:45 smithi116 ceph-mon[27165]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T04:16:45.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:45 smithi022 ceph-mon[24756]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T04:16:47.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:47 smithi116 ceph-mon[27165]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-24T04:16:47.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:47 smithi022 ceph-mon[24756]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-24T04:16:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:49 smithi116 ceph-mon[27165]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T04:16:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:16:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:49.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:49 smithi022 ceph-mon[24756]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T04:16:49.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:16:49.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:50.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:50 smithi116 ceph-mon[27165]: Deploying daemon osd.5 on smithi116 2024-03-24T04:16:50.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:50 smithi022 ceph-mon[24756]: Deploying daemon osd.5 on smithi116 2024-03-24T04:16:51.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:51 smithi116 ceph-mon[27165]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:51 smithi022 ceph-mon[24756]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:53.208 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:53 smithi116 ceph-mon[27165]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:53.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:53 smithi022 ceph-mon[24756]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:54 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:54.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:54 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:54.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:54 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:54.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:54 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:54.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:54 smithi022 ceph-mon[24756]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:54 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:54 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:16:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:54 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:16:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:54 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:16:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:54 smithi116 ceph-mon[27165]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:56 smithi116 ceph-mon[27165]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:56.608 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 5 on host 'smithi116' 2024-03-24T04:16:56.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:56 smithi022 ceph-mon[24756]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:57 smithi116 ceph-mon[27165]: from='osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:16:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:57 smithi116 ceph-mon[27165]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:16:57.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:57 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:57.487 DEBUG:teuthology.orchestra.run.smithi116:osd.5> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T04:16:57.490 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-24T04:16:57.490 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T04:16:57.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:57 smithi022 ceph-mon[24756]: from='osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:16:57.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:57 smithi022 ceph-mon[24756]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:16:57.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:57 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:58.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:58.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T04:16:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T04:16:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: from='osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:58.474 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[34397]: 2024-03-24T04:16:58.344+0000 7fe5924a7700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T04:16:58.474 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:16:58 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[34397]: 2024-03-24T04:16:58.349+0000 7fe58ae3a700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:16:58.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: from='osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:16:58.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:58 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319] boot 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T04:16:59.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:16:59 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130646 s, 803 MB/s 2024-03-24T04:16:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T04:16:59.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:16:59.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T04:16:59.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:59.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:16:59.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: osd.5 [v2:172.21.15.116:6808/2235902319,v1:172.21.15.116:6809/2235902319] boot 2024-03-24T04:16:59.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T04:16:59.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:16:59 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:17:00.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:00 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:17:00.457 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:00 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:17:00.458 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:00 smithi116 ceph-mon[27165]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:17:00.458 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:00 smithi116 ceph-mon[27165]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T04:17:00.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:00 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:17:00.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:00 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:17:00.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:00 smithi022 ceph-mon[24756]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T04:17:00.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:00 smithi022 ceph-mon[24756]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T04:17:00.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-24T04:17:02.512 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:02 smithi116 ceph-mon[27165]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:02.512 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:02 smithi116 ceph-mon[27165]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T04:17:02.512 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:02 smithi116 ceph-mon[27165]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T04:17:02.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:02 smithi022 ceph-mon[24756]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:02.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:02 smithi022 ceph-mon[24756]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T04:17:02.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:02 smithi022 ceph-mon[24756]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi116 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi116 to 7968M 2024-03-24T04:17:03.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:03 smithi022 ceph-mon[24756]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T04:17:03.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi116 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi116 to 7968M 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:03.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:03.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:03.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:03.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:03 smithi116 ceph-mon[27165]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T04:17:04.882 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:04 smithi116 ceph-mon[27165]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:04.883 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:04 smithi116 ceph-mon[27165]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:17:04.883 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:04 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:17:04.883 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:04 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:17:04.883 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:04 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:05.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:04 smithi022 ceph-mon[24756]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:05.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:04 smithi022 ceph-mon[24756]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:17:05.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:04 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:17:05.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:04 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:17:05.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:04 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:06.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:06.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/3037388341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]: dispatch 2024-03-24T04:17:06.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]: dispatch 2024-03-24T04:17:06.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]': finished 2024-03-24T04:17:06.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T04:17:06.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:06.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:06 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1652736069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:17:07.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/3037388341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]: dispatch 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]: dispatch 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc9878e-7af2-42dd-961f-0ef01d78c3a7"}]': finished 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:07.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:06 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1652736069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:17:08.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:08 smithi116 ceph-mon[27165]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:09.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:08 smithi022 ceph-mon[24756]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:10.699 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:10 smithi116 ceph-mon[27165]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:10.849 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:10 smithi022 ceph-mon[24756]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:12.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:12 smithi116 ceph-mon[27165]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:12.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:12 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:17:12.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:12 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:12.657 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:12 smithi116 ceph-mon[27165]: Deploying daemon osd.6 on smithi116 2024-03-24T04:17:13.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:12 smithi022 ceph-mon[24756]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:13.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:12 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:17:13.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:12 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:13.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:12 smithi022 ceph-mon[24756]: Deploying daemon osd.6 on smithi116 2024-03-24T04:17:14.969 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:14 smithi116 ceph-mon[27165]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:15.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:14 smithi022 ceph-mon[24756]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:16.811 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:16 smithi116 ceph-mon[27165]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:17.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:16 smithi022 ceph-mon[24756]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:18.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:17 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:18.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:17 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:18.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:17 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:18.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:17 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:18.118 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:17 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:18.118 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:17 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:18.118 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:17 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:18.119 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:17 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:19.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:18 smithi022 ceph-mon[24756]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:19.199 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:18 smithi116 ceph-mon[27165]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:19.650 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-24T04:17:20.355 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T04:17:20.357 INFO:tasks.cephadm:Deploying osd.7 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-24T04:17:20.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T04:17:20.943 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:20 smithi116 ceph-mon[27165]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:20.943 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:20.943 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:20 smithi116 ceph-mon[27165]: from='osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:17:20.943 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:20 smithi116 ceph-mon[27165]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:17:21.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:20 smithi022 ceph-mon[24756]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:21.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:21.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:20 smithi022 ceph-mon[24756]: from='osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:17:21.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:20 smithi022 ceph-mon[24756]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:17:21.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-mon[27165]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T04:17:21.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-mon[27165]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T04:17:21.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:21.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-mon[27165]: from='osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:21.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-mon[27165]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:21.974 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[37393]: 2024-03-24T04:17:21.764+0000 7f461e64e700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T04:17:21.974 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:17:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[37393]: 2024-03-24T04:17:21.770+0000 7f46157de700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:17:22.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:21 smithi022 ceph-mon[24756]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T04:17:22.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:21 smithi022 ceph-mon[24756]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T04:17:22.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:21 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:22.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:21 smithi022 ceph-mon[24756]: from='osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:22.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:21 smithi022 ceph-mon[24756]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:22.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:22.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:17:22.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T04:17:22.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:22.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:22.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533] boot 2024-03-24T04:17:22.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T04:17:22.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:22.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:22.888 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T04:17:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T04:17:22.889 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T04:17:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T04:17:22.889 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124448 s, 843 MB/s 2024-03-24T04:17:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T04:17:23.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T04:17:23.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:17:23.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: osd.6 [v2:172.21.15.116:6816/620836533,v1:172.21.15.116:6817/620836533] boot 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:17:23.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:23.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-24T04:17:23.842 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:23 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:17:23.842 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:23 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:17:23.842 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:23 smithi116 ceph-mon[27165]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T04:17:24.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:23 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:17:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:23 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:17:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:23 smithi022 ceph-mon[24756]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T04:17:24.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:24 smithi116 ceph-mon[27165]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:24.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:24 smithi116 ceph-mon[27165]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T04:17:25.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:24 smithi022 ceph-mon[24756]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:25.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:24 smithi022 ceph-mon[24756]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T04:17:26.449 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:26 smithi116 ceph-mon[27165]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:26.450 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:26 smithi116 ceph-mon[27165]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T04:17:26.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:26 smithi022 ceph-mon[24756]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:26.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:26 smithi022 ceph-mon[24756]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T04:17:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:27 smithi116 ceph-mon[27165]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T04:17:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:27 smithi116 ceph-mon[27165]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:17:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:17:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:17:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:27.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:27 smithi022 ceph-mon[24756]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T04:17:27.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:27 smithi022 ceph-mon[24756]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:17:27.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T04:17:27.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T04:17:27.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi116 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi116 to 5312M 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:29.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:29.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:29.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:28 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi116 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:29.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi116 to 5312M 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:29.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:28 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:30.285 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/2631228069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]: dispatch 2024-03-24T04:17:30.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]: dispatch 2024-03-24T04:17:30.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]': finished 2024-03-24T04:17:30.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T04:17:30.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:30.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:29 smithi116 ceph-mon[27165]: from='client.? 172.21.15.116:0/1597585206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:17:30.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/2631228069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]: dispatch 2024-03-24T04:17:30.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]: dispatch 2024-03-24T04:17:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98fae651-60e1-4e26-b16f-3ddf93eba6a1"}]': finished 2024-03-24T04:17:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T04:17:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:29 smithi022 ceph-mon[24756]: from='client.? 172.21.15.116:0/1597585206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T04:17:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:30 smithi022 ceph-mon[24756]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:30 smithi116 ceph-mon[27165]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:33.286 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:32 smithi116 ceph-mon[27165]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:33.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:32 smithi022 ceph-mon[24756]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:35.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:34 smithi022 ceph-mon[24756]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T04:17:35.146 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:34 smithi116 ceph-mon[27165]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T04:17:36.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:35 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:17:36.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:35 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:35 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:17:36.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:35 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:37.106 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:36 smithi116 ceph-mon[27165]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T04:17:37.106 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:36 smithi116 ceph-mon[27165]: Deploying daemon osd.7 on smithi116 2024-03-24T04:17:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:36 smithi022 ceph-mon[24756]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T04:17:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:36 smithi022 ceph-mon[24756]: Deploying daemon osd.7 on smithi116 2024-03-24T04:17:39.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:38 smithi022 ceph-mon[24756]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T04:17:39.353 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:38 smithi116 ceph-mon[27165]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T04:17:41.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:40 smithi116 ceph-mon[27165]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:41.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:40 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:41.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:40 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:41.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:40 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:41.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:40 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:41.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:40 smithi022 ceph-mon[24756]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:41.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:40 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:41.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:40 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:41.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:40 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:41.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:40 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:43.201 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:42 smithi116 ceph-mon[27165]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:43.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:42 smithi022 ceph-mon[24756]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:43.835 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 7 on host 'smithi116' 2024-03-24T04:17:44.131 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:17:44.131 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:17:44.131 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:44 smithi116 ceph-mon[27165]: from='osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:17:44.131 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:44 smithi116 ceph-mon[27165]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:17:44.131 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:17:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:17:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:44 smithi022 ceph-mon[24756]: from='osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:17:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:44 smithi022 ceph-mon[24756]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:17:44.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:44.420 DEBUG:teuthology.orchestra.run.smithi116:osd.7> sudo journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T04:17:44.423 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T04:17:44.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: from='osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:45.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:45 smithi022 ceph-mon[24756]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:45.390 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:45.390 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T04:17:45.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T04:17:45.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:45.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: from='osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:45.391 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-mon[27165]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:17:45.391 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[40405]: 2024-03-24T04:17:45.015+0000 7f71a457d700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T04:17:45.391 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:17:45 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[40405]: 2024-03-24T04:17:45.023+0000 7f719ef14700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:17:46.012 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:17:46.012 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T04:17:46.070 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T04:17:46.070 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312] boot 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T04:17:46.071 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:46 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: osd.7 [v2:172.21.15.116:6824/4149613312,v1:172.21.15.116:6825/4149613312] boot 2024-03-24T04:17:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T04:17:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:46 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:17:46.775 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:17:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:47 smithi116 ceph-mon[27165]: purged_snaps scrub starts 2024-03-24T04:17:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:47 smithi116 ceph-mon[27165]: purged_snaps scrub ok 2024-03-24T04:17:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:47 smithi116 ceph-mon[27165]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:47 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1758488330' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T04:17:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:47 smithi116 ceph-mon[27165]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T04:17:47.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:47 smithi022 ceph-mon[24756]: purged_snaps scrub starts 2024-03-24T04:17:47.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:47 smithi022 ceph-mon[24756]: purged_snaps scrub ok 2024-03-24T04:17:47.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:47 smithi022 ceph-mon[24756]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T04:17:47.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:47 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1758488330' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T04:17:47.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:47 smithi022 ceph-mon[24756]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T04:17:47.324 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711253866,"num_in_osds":8,"osd_in_since":1711253849,"num_remapped_pgs":1} 2024-03-24T04:17:47.325 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T04:17:47.542 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:17:49.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:17:49.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T04:17:49.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:49.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.471 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:17:49.471 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":58,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","created":"2024-03-24T04:11:48.000687+0000","modified":"2024-03-24T04:17:49.073321+0000","last_up_change":"2024-03-24T04:17:46.005459+0000","last_in_change":"2024-03-24T04:17:29.340024+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T04:15:47.566585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f12be2f-9abd-44d0-9be5-d478e00bf4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6803","nonce":1971225723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6805","nonce":1971225723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6809","nonce":1971225723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6807","nonce":1971225723}]},"public_addr":"172.21.15.22:6803/1971225723","cluster_addr":"172.21.15.22:6805/1971225723","heartbeat_back_addr":"172.21.15.22:6809/1971225723","heartbeat_front_addr":"172.21.15.22:6807/1971225723","state":["exists","up"]},{"osd":1,"uuid":"cc5da2b7-2441-4bec-98be-21c5dd13f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6811","nonce":2734658854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6813","nonce":2734658854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6817","nonce":2734658854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6815","nonce":2734658854}]},"public_addr":"172.21.15.22:6811/2734658854","cluster_addr":"172.21.15.22:6813/2734658854","heartbeat_back_addr":"172.21.15.22:6817/2734658854","heartbeat_front_addr":"172.21.15.22:6815/2734658854","state":["exists","up"]},{"osd":2,"uuid":"5a27c891-0e0b-4c8b-8404-334ff29c44f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6819","nonce":3565337546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6821","nonce":3565337546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6825","nonce":3565337546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6823","nonce":3565337546}]},"public_addr":"172.21.15.22:6819/3565337546","cluster_addr":"172.21.15.22:6821/3565337546","heartbeat_back_addr":"172.21.15.22:6825/3565337546","heartbeat_front_addr":"172.21.15.22:6823/3565337546","state":["exists","up"]},{"osd":3,"uuid":"5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6827","nonce":2272173823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6829","nonce":2272173823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6833","nonce":2272173823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6831","nonce":2272173823}]},"public_addr":"172.21.15.22:6827/2272173823","cluster_addr":"172.21.15.22:6829/2272173823","heartbeat_back_addr":"172.21.15.22:6833/2272173823","heartbeat_front_addr":"172.21.15.22:6831/2272173823","state":["exists","up"]},{"osd":4,"uuid":"1de55b84-e20c-434a-8890-25b0a56e2093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6801","nonce":3256324323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6803","nonce":3256324323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6807","nonce":3256324323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6805","nonce":3256324323}]},"public_addr":"172.21.15.116:6801/3256324323","cluster_addr":"172.21.15.116:6803/3256324323","heartbeat_back_addr":"172.21.15.116:6807/3256324323","heartbeat_front_addr":"172.21.15.116:6805/3256324323","state":["exists","up"]},{"osd":5,"uuid":"1b426fc4-8e82-402b-90ca-dfb3bea82176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6809","nonce":2235902319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6811","nonce":2235902319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6815","nonce":2235902319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6813","nonce":2235902319}]},"public_addr":"172.21.15.116:6809/2235902319","cluster_addr":"172.21.15.116:6811/2235902319","heartbeat_back_addr":"172.21.15.116:6815/2235902319","heartbeat_front_addr":"172.21.15.116:6813/2235902319","state":["exists","up"]},{"osd":6,"uuid":"edc9878e-7af2-42dd-961f-0ef01d78c3a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6817","nonce":620836533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6819","nonce":620836533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6823","nonce":620836533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6821","nonce":620836533}]},"public_addr":"172.21.15.116:6817/620836533","cluster_addr":"172.21.15.116:6819/620836533","heartbeat_back_addr":"172.21.15.116:6823/620836533","heartbeat_front_addr":"172.21.15.116:6821/620836533","state":["exists","up"]},{"osd":7,"uuid":"98fae651-60e1-4e26-b16f-3ddf93eba6a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6825","nonce":4149613312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6827","nonce":4149613312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6831","nonce":4149613312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6829","nonce":4149613312}]},"public_addr":"172.21.15.116:6825/4149613312","cluster_addr":"172.21.15.116:6827/4149613312","heartbeat_back_addr":"172.21.15.116:6831/4149613312","heartbeat_front_addr":"172.21.15.116:6829/4149613312","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:14:49.823435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:17.066096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:45.063408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:11.739317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:34.098249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:57.390083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:20.825502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:44.732076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3543408237":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/4145413097":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/234367322":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6800/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/2013097742":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:6800/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/916978414":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/419759058":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/2584741775":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6801/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/4175807310":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6800/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/3708120614":"2024-03-25T04:12:19.223857+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T04:17:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:17:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:17:49.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:17:50.189 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T04:15:47.566585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-24T04:17:50.190 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T04:17:50.414 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:17:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:50 smithi116 ceph-mon[27165]: Detected new or changed devices on smithi116 2024-03-24T04:17:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:50 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi116 to 3984M 2024-03-24T04:17:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:50 smithi116 ceph-mon[27165]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T04:17:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:50 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/266488506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:17:50.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:50 smithi022 ceph-mon[24756]: Detected new or changed devices on smithi116 2024-03-24T04:17:50.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:50 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi116 to 3984M 2024-03-24T04:17:50.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:50 smithi022 ceph-mon[24756]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T04:17:50.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:50 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/266488506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:17:51.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:51 smithi022 ceph-mon[24756]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:17:51.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:51 smithi022 ceph-mon[24756]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T04:17:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:51 smithi116 ceph-mon[27165]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:17:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:51 smithi116 ceph-mon[27165]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T04:17:52.122 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T04:17:52.836 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T04:17:52.836 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T04:17:53.396 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:53 smithi022 ceph-mon[24756]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-24T04:17:53.396 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:53 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3877964775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T04:17:53.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:53 smithi116 ceph-mon[27165]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-24T04:17:53.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:53 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/3877964775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T04:17:54.802 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-24T04:17:54.802 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQByqf9lacqdLxAAf6Alan+OnBTKKER+03eWQg== 2024-03-24T04:17:55.457 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:55 smithi022 ceph-mon[24756]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:55.457 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:55 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4245733623' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T04:17:55.457 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:55 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4245733623' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T04:17:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:55 smithi116 ceph-mon[27165]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T04:17:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:55 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4245733623' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T04:17:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:55 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4245733623' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T04:17:55.480 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T04:17:55.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T04:17:55.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T04:17:55.522 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T04:17:55.522 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T04:17:55.522 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T04:17:55.778 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:17:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:57 smithi116 ceph-mon[27165]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T04:17:57.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:57 smithi022 ceph-mon[24756]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-24T04:17:57.568 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:17:58.282 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi022.ijpjvk","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3908342254},{"type":"v1","addr":"172.21.15.22:6801","nonce":3908342254}]},"active_addr":"172.21.15.22:6801/3908342254","active_change":"2024-03-24T04:13:43.054004+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14222,"name":"smithi116.oxktjt","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.22:8443/","prometheus":"http://172.21.15.22:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":1333550415}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3306740224}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":4229620390}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":952602130}]}]}} 2024-03-24T04:17:58.290 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T04:17:58.290 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T04:17:58.291 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T04:17:58.514 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:17:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:58 smithi022 ceph-mon[24756]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T04:17:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:17:58 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4007892835' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T04:17:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:58 smithi116 ceph-mon[27165]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T04:17:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:17:58 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4007892835' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T04:18:00.289 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:18:00.289 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":59,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","created":"2024-03-24T04:11:48.000687+0000","modified":"2024-03-24T04:17:50.075397+0000","last_up_change":"2024-03-24T04:17:46.005459+0000","last_in_change":"2024-03-24T04:17:29.340024+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T04:15:47.566585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f12be2f-9abd-44d0-9be5-d478e00bf4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6803","nonce":1971225723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6805","nonce":1971225723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6809","nonce":1971225723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6807","nonce":1971225723}]},"public_addr":"172.21.15.22:6803/1971225723","cluster_addr":"172.21.15.22:6805/1971225723","heartbeat_back_addr":"172.21.15.22:6809/1971225723","heartbeat_front_addr":"172.21.15.22:6807/1971225723","state":["exists","up"]},{"osd":1,"uuid":"cc5da2b7-2441-4bec-98be-21c5dd13f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6811","nonce":2734658854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6813","nonce":2734658854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6817","nonce":2734658854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6815","nonce":2734658854}]},"public_addr":"172.21.15.22:6811/2734658854","cluster_addr":"172.21.15.22:6813/2734658854","heartbeat_back_addr":"172.21.15.22:6817/2734658854","heartbeat_front_addr":"172.21.15.22:6815/2734658854","state":["exists","up"]},{"osd":2,"uuid":"5a27c891-0e0b-4c8b-8404-334ff29c44f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6819","nonce":3565337546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6821","nonce":3565337546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6825","nonce":3565337546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6823","nonce":3565337546}]},"public_addr":"172.21.15.22:6819/3565337546","cluster_addr":"172.21.15.22:6821/3565337546","heartbeat_back_addr":"172.21.15.22:6825/3565337546","heartbeat_front_addr":"172.21.15.22:6823/3565337546","state":["exists","up"]},{"osd":3,"uuid":"5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6827","nonce":2272173823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6829","nonce":2272173823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6833","nonce":2272173823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6831","nonce":2272173823}]},"public_addr":"172.21.15.22:6827/2272173823","cluster_addr":"172.21.15.22:6829/2272173823","heartbeat_back_addr":"172.21.15.22:6833/2272173823","heartbeat_front_addr":"172.21.15.22:6831/2272173823","state":["exists","up"]},{"osd":4,"uuid":"1de55b84-e20c-434a-8890-25b0a56e2093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6801","nonce":3256324323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6803","nonce":3256324323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6807","nonce":3256324323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6805","nonce":3256324323}]},"public_addr":"172.21.15.116:6801/3256324323","cluster_addr":"172.21.15.116:6803/3256324323","heartbeat_back_addr":"172.21.15.116:6807/3256324323","heartbeat_front_addr":"172.21.15.116:6805/3256324323","state":["exists","up"]},{"osd":5,"uuid":"1b426fc4-8e82-402b-90ca-dfb3bea82176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6809","nonce":2235902319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6811","nonce":2235902319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6815","nonce":2235902319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6813","nonce":2235902319}]},"public_addr":"172.21.15.116:6809/2235902319","cluster_addr":"172.21.15.116:6811/2235902319","heartbeat_back_addr":"172.21.15.116:6815/2235902319","heartbeat_front_addr":"172.21.15.116:6813/2235902319","state":["exists","up"]},{"osd":6,"uuid":"edc9878e-7af2-42dd-961f-0ef01d78c3a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6817","nonce":620836533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6819","nonce":620836533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6823","nonce":620836533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6821","nonce":620836533}]},"public_addr":"172.21.15.116:6817/620836533","cluster_addr":"172.21.15.116:6819/620836533","heartbeat_back_addr":"172.21.15.116:6823/620836533","heartbeat_front_addr":"172.21.15.116:6821/620836533","state":["exists","up"]},{"osd":7,"uuid":"98fae651-60e1-4e26-b16f-3ddf93eba6a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6825","nonce":4149613312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6827","nonce":4149613312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6831","nonce":4149613312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6829","nonce":4149613312}]},"public_addr":"172.21.15.116:6825/4149613312","cluster_addr":"172.21.15.116:6827/4149613312","heartbeat_back_addr":"172.21.15.116:6831/4149613312","heartbeat_front_addr":"172.21.15.116:6829/4149613312","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:14:49.823435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:17.066096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:45.063408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:11.739317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:34.098249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:57.390083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:20.825502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:44.732076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3543408237":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/4145413097":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/234367322":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6800/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/2013097742":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:6800/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/916978414":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/419759058":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/2584741775":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6801/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/4175807310":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6800/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/3708120614":"2024-03-25T04:12:19.223857+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T04:18:00.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:00 smithi022 ceph-mon[24756]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T04:18:00.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:00 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3184860803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:18:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:00 smithi116 ceph-mon[27165]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T04:18:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:00 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/3184860803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:18:00.896 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T04:18:00.896 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T04:18:01.120 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:02.384 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:02 smithi022 ceph-mon[24756]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-03-24T04:18:02.678 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:02 smithi116 ceph-mon[27165]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-03-24T04:18:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:18:02.771 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":59,"fsid":"8f0be64c-e994-11ee-95cd-87774f69a715","created":"2024-03-24T04:11:48.000687+0000","modified":"2024-03-24T04:17:50.075397+0000","last_up_change":"2024-03-24T04:17:46.005459+0000","last_in_change":"2024-03-24T04:17:29.340024+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T04:15:47.566585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f12be2f-9abd-44d0-9be5-d478e00bf4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6803","nonce":1971225723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6805","nonce":1971225723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6809","nonce":1971225723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":1971225723},{"type":"v1","addr":"172.21.15.22:6807","nonce":1971225723}]},"public_addr":"172.21.15.22:6803/1971225723","cluster_addr":"172.21.15.22:6805/1971225723","heartbeat_back_addr":"172.21.15.22:6809/1971225723","heartbeat_front_addr":"172.21.15.22:6807/1971225723","state":["exists","up"]},{"osd":1,"uuid":"cc5da2b7-2441-4bec-98be-21c5dd13f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6811","nonce":2734658854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6813","nonce":2734658854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6817","nonce":2734658854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":2734658854},{"type":"v1","addr":"172.21.15.22:6815","nonce":2734658854}]},"public_addr":"172.21.15.22:6811/2734658854","cluster_addr":"172.21.15.22:6813/2734658854","heartbeat_back_addr":"172.21.15.22:6817/2734658854","heartbeat_front_addr":"172.21.15.22:6815/2734658854","state":["exists","up"]},{"osd":2,"uuid":"5a27c891-0e0b-4c8b-8404-334ff29c44f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6819","nonce":3565337546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6821","nonce":3565337546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6825","nonce":3565337546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3565337546},{"type":"v1","addr":"172.21.15.22:6823","nonce":3565337546}]},"public_addr":"172.21.15.22:6819/3565337546","cluster_addr":"172.21.15.22:6821/3565337546","heartbeat_back_addr":"172.21.15.22:6825/3565337546","heartbeat_front_addr":"172.21.15.22:6823/3565337546","state":["exists","up"]},{"osd":3,"uuid":"5f3fc4dd-c4ae-4fd1-b17e-0dbf23249b69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6827","nonce":2272173823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6829","nonce":2272173823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6833","nonce":2272173823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2272173823},{"type":"v1","addr":"172.21.15.22:6831","nonce":2272173823}]},"public_addr":"172.21.15.22:6827/2272173823","cluster_addr":"172.21.15.22:6829/2272173823","heartbeat_back_addr":"172.21.15.22:6833/2272173823","heartbeat_front_addr":"172.21.15.22:6831/2272173823","state":["exists","up"]},{"osd":4,"uuid":"1de55b84-e20c-434a-8890-25b0a56e2093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6801","nonce":3256324323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6803","nonce":3256324323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6807","nonce":3256324323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3256324323},{"type":"v1","addr":"172.21.15.116:6805","nonce":3256324323}]},"public_addr":"172.21.15.116:6801/3256324323","cluster_addr":"172.21.15.116:6803/3256324323","heartbeat_back_addr":"172.21.15.116:6807/3256324323","heartbeat_front_addr":"172.21.15.116:6805/3256324323","state":["exists","up"]},{"osd":5,"uuid":"1b426fc4-8e82-402b-90ca-dfb3bea82176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6809","nonce":2235902319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6811","nonce":2235902319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6815","nonce":2235902319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2235902319},{"type":"v1","addr":"172.21.15.116:6813","nonce":2235902319}]},"public_addr":"172.21.15.116:6809/2235902319","cluster_addr":"172.21.15.116:6811/2235902319","heartbeat_back_addr":"172.21.15.116:6815/2235902319","heartbeat_front_addr":"172.21.15.116:6813/2235902319","state":["exists","up"]},{"osd":6,"uuid":"edc9878e-7af2-42dd-961f-0ef01d78c3a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6817","nonce":620836533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6819","nonce":620836533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6823","nonce":620836533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":620836533},{"type":"v1","addr":"172.21.15.116:6821","nonce":620836533}]},"public_addr":"172.21.15.116:6817/620836533","cluster_addr":"172.21.15.116:6819/620836533","heartbeat_back_addr":"172.21.15.116:6823/620836533","heartbeat_front_addr":"172.21.15.116:6821/620836533","state":["exists","up"]},{"osd":7,"uuid":"98fae651-60e1-4e26-b16f-3ddf93eba6a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6825","nonce":4149613312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6827","nonce":4149613312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6831","nonce":4149613312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4149613312},{"type":"v1","addr":"172.21.15.116:6829","nonce":4149613312}]},"public_addr":"172.21.15.116:6825/4149613312","cluster_addr":"172.21.15.116:6827/4149613312","heartbeat_back_addr":"172.21.15.116:6831/4149613312","heartbeat_front_addr":"172.21.15.116:6829/4149613312","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:14:49.823435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:17.066096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:15:45.063408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:11.739317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:34.098249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:16:57.390083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:20.825502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T04:17:44.732076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3543408237":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/4145413097":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/234367322":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6800/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:0/2013097742":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:6800/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/916978414":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/419759058":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/2584741775":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6801/2296714771":"2024-03-25T04:12:48.577030+0000","172.21.15.22:6801/1580053261":"2024-03-25T04:12:19.223857+0000","172.21.15.22:0/4175807310":"2024-03-25T04:12:19.223857+0000","172.21.15.22:6800/865697305":"2024-03-25T04:13:43.053727+0000","172.21.15.22:0/3708120614":"2024-03-25T04:12:19.223857+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T04:18:03.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:03 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1787092993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:18:03.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T04:18:03.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T04:18:03.440 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T04:18:03.440 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T04:18:03.441 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T04:18:03.441 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T04:18:03.442 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T04:18:03.442 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T04:18:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:03 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1787092993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T04:18:03.863 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.868 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.869 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.869 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.869 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.870 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.878 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:03.905 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:04.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:04 smithi022 ceph-mon[24756]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:04 smithi116 ceph-mon[27165]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:06.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:06 smithi022 ceph-mon[24756]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:06 smithi116 ceph-mon[27165]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:08.642 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:08 smithi022 ceph-mon[24756]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:08 smithi116 ceph-mon[27165]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:10.086 INFO:teuthology.orchestra.run.smithi022.stdout:201863462923 2024-03-24T04:18:10.086 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T04:18:10.623 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:10 smithi022 ceph-mon[24756]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:10 smithi116 ceph-mon[27165]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:12 smithi116 ceph-mon[27165]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:12 smithi022 ceph-mon[24756]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:14.575 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:14 smithi022 ceph-mon[24756]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:14 smithi116 ceph-mon[27165]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:15.330 INFO:teuthology.orchestra.run.smithi022.stdout:81604378657 2024-03-24T04:18:15.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T04:18:16.519 INFO:teuthology.orchestra.run.smithi022.stdout:38654705707 2024-03-24T04:18:16.519 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T04:18:16.594 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:16 smithi022 ceph-mon[24756]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:16.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:16 smithi116 ceph-mon[27165]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:17.384 INFO:teuthology.orchestra.run.smithi022.stdout:167503724562 2024-03-24T04:18:17.385 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T04:18:18.097 INFO:teuthology.orchestra.run.smithi022.stdout:60129542183 2024-03-24T04:18:18.098 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T04:18:18.370 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:18 smithi022 ceph-mon[24756]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:18 smithi116 ceph-mon[27165]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:20.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:20 smithi022 ceph-mon[24756]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:20.627 INFO:teuthology.orchestra.run.smithi022.stdout:133143986200 2024-03-24T04:18:20.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T04:18:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:20 smithi116 ceph-mon[27165]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:21.363 INFO:teuthology.orchestra.run.smithi022.stdout:111669149725 2024-03-24T04:18:21.363 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T04:18:21.409 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:21.412 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:21.413 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:21.757 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:21.758 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:22.613 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:22 smithi022 ceph-mon[24756]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:22 smithi116 ceph-mon[27165]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:24 smithi022 ceph-mon[24756]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:24.648 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:24 smithi116 ceph-mon[27165]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:26.647 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:26 smithi022 ceph-mon[24756]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:26.676 INFO:teuthology.orchestra.run.smithi022.stdout:236223201289 2024-03-24T04:18:26.676 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T04:18:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:26 smithi116 ceph-mon[27165]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:27.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:27 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2401080862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T04:18:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:27 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2401080862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T04:18:28.601 INFO:teuthology.orchestra.run.smithi022.stdout:201863462926 2024-03-24T04:18:28.601 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:28 smithi022 ceph-mon[24756]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:28 smithi116 ceph-mon[27165]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:30.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462923 got 201863462926 for osd.6 2024-03-24T04:18:30.163 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:30.388 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:30 smithi022 ceph-mon[24756]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:30 smithi116 ceph-mon[27165]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:30.771 INFO:teuthology.orchestra.run.smithi022.stdout:81604378660 2024-03-24T04:18:30.978 INFO:teuthology.orchestra.run.smithi022.stdout:60129542185 2024-03-24T04:18:31.358 INFO:teuthology.orchestra.run.smithi022.stdout:133143986202 2024-03-24T04:18:31.615 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:31 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1433591281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T04:18:31.615 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:31 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3416928558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T04:18:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:31 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1433591281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T04:18:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:31 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/3416928558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T04:18:32.192 INFO:teuthology.orchestra.run.smithi022.stdout:111669149727 2024-03-24T04:18:32.372 INFO:teuthology.orchestra.run.smithi022.stdout:167503724565 2024-03-24T04:18:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:32 smithi022 ceph-mon[24756]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:32 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4291651229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T04:18:32.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:32 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4089468722' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T04:18:32.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:32 smithi116 ceph-mon[27165]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:32.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:32 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4291651229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T04:18:32.679 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:32 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4089468722' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T04:18:32.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724562 got 167503724565 for osd.5 2024-03-24T04:18:32.943 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:33.279 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:33.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:33 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/694318949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T04:18:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:33 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/694318949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T04:18:34.384 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986202 for osd.4 2024-03-24T04:18:34.385 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:34 smithi116 ceph-mon[27165]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:34.759 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:34 smithi022 ceph-mon[24756]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:34.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378660 for osd.2 2024-03-24T04:18:34.801 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:35.429 INFO:teuthology.orchestra.run.smithi022.stdout:38654705712 2024-03-24T04:18:35.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149725 got 111669149727 for osd.3 2024-03-24T04:18:35.587 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:36.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705707 got 38654705712 for osd.0 2024-03-24T04:18:36.192 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:36.444 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:36 smithi022 ceph-mon[24756]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:36.444 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:36 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2741913964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T04:18:36.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:36 smithi116 ceph-mon[27165]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:36.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:36 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2741913964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T04:18:37.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542185 for osd.1 2024-03-24T04:18:37.041 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:37.739 INFO:teuthology.orchestra.run.smithi022.stdout:236223201293 2024-03-24T04:18:38.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201289 got 236223201293 for osd.7 2024-03-24T04:18:38.338 DEBUG:teuthology.parallel:result is None 2024-03-24T04:18:38.339 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T04:18:38.339 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T04:18:38.561 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:38.683 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:38 smithi022 ceph-mon[24756]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:38.683 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:38 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3758670724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T04:18:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:38 smithi116 ceph-mon[27165]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:38 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/3758670724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T04:18:40.326 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:18:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T04:18:40.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:40 smithi022 ceph-mon[24756]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:40 smithi116 ceph-mon[27165]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:41.017 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-03-24T04:18:39.296792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48800,"kb_used_data":6304,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912416,"statfs":{"total":767960285184,"available":767910313984,"internally_reserved":0,"allocated":6455296,"data_stored":3546488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002424"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-24T04:17:50.084575+0000","last_change":"2024-03-24T04:17:50.084575+0000","last_active":"2024-03-24T04:17:50.084575+0000","last_peered":"2024-03-24T04:17:50.084575+0000","last_clean":"2024-03-24T04:17:50.084575+0000","last_became_active":"2024-03-24T04:17:50.084133+0000","last_became_peered":"2024-03-24T04:17:50.084133+0000","last_unstale":"2024-03-24T04:17:50.084575+0000","last_undegraded":"2024-03-24T04:17:50.084575+0000","last_fullsized":"2024-03-24T04:17:50.084575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T04:15:48.117768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T04:15:48.117768+0000","last_clean_scrub_stamp":"2024-03-24T04:15:48.117768+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:51.535126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6112,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739040,"statfs":{"total":95995035648,"available":95988776960,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.495}]},{"osd":1,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.45000000000000001}]},{"osd":2,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52000000000000002}]},{"osd":3,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.376},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.436},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.31},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.34100000000000003,"15min":0.315},"min":{"1min":0.29799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 24 04:17:50 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.376,"15min":0.373},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.374}]},{"osd":3,"last update":"Sun Mar 24 04:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.44}]},{"osd":4,"last update":"Sun Mar 24 04:17:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.749}]},{"osd":5,"last update":"Sun Mar 24 04:17:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.71999999999999997}]},{"osd":6,"last update":"Sun Mar 24 04:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 04:18:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.30599999999999999},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Mar 24 04:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.32400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sun Mar 24 04:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42199999999999999,"15min":0.42399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.442,"15min":0.44500000000000001},"min":{"1min":0.28799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42899999999999999}]},{"osd":4,"last update":"Sun Mar 24 04:17:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65200000000000002}]},{"osd":5,"last update":"Sun Mar 24 04:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.81000000000000005}]},{"osd":6,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6232,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738920,"statfs":{"total":95995035648,"available":95988654080,"internally_reserved":0,"allocated":606208,"data_stored":244572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:17:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.371,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Mar 24 04:17:48 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38700000000000001,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.38400000000000001,"15min":0.36899999999999999},"min":{"1min":0.28299999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27200000000000002}]},{"osd":3,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.312},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.26300000000000001}]},{"osd":4,"last update":"Sun Mar 24 04:17:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55600000000000005}]},{"osd":5,"last update":"Sun Mar 24 04:18:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5716,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739436,"statfs":{"total":95995035648,"available":95989182464,"internally_reserved":0,"allocated":602112,"data_stored":244645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42299999999999999,"15min":0.42799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.32800000000000001}]},{"osd":2,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48999999999999999,"15min":0.496},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.311}]},{"osd":4,"last update":"Sun Mar 24 04:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.443}]},{"osd":5,"last update":"Sun Mar 24 04:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.45200000000000001}]},{"osd":6,"last update":"Sun Mar 24 04:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.65200000000000002}]},{"osd":2,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67200000000000004}]},{"osd":3,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Mar 24 04:17:59 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.47399999999999998}]},{"osd":6,"last update":"Sun Mar 24 04:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.621},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724566,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45200000000000001}]},{"osd":1,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.436}]},{"osd":2,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.41399999999999998}]},{"osd":3,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.51100000000000001}]},{"osd":6,"last update":"Sun Mar 24 04:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T04:18:41.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T04:18:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:41.618 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:41 smithi022 ceph-mon[24756]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:18:41.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:41 smithi116 ceph-mon[27165]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:18:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:42 smithi116 ceph-mon[27165]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:42.671 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:42 smithi022 ceph-mon[24756]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:18:43.099 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T04:18:43.666 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:43 smithi022 ceph-mon[24756]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:18:43.666 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:18:43.667 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:18:43.691 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-03-24T04:18:41.297126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48800,"kb_used_data":6304,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912416,"statfs":{"total":767960285184,"available":767910313984,"internally_reserved":0,"allocated":6455296,"data_stored":3546488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002273"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-24T04:17:50.084575+0000","last_change":"2024-03-24T04:17:50.084575+0000","last_active":"2024-03-24T04:17:50.084575+0000","last_peered":"2024-03-24T04:17:50.084575+0000","last_clean":"2024-03-24T04:17:50.084575+0000","last_became_active":"2024-03-24T04:17:50.084133+0000","last_became_peered":"2024-03-24T04:17:50.084133+0000","last_unstale":"2024-03-24T04:17:50.084575+0000","last_undegraded":"2024-03-24T04:17:50.084575+0000","last_fullsized":"2024-03-24T04:17:50.084575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T04:15:48.117768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T04:15:48.117768+0000","last_clean_scrub_stamp":"2024-03-24T04:15:48.117768+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:51.535126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":6,"up_from":47,"seq":201863462929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6112,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739040,"statfs":{"total":95995035648,"available":95988776960,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.495}]},{"osd":1,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.45000000000000001}]},{"osd":2,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52000000000000002}]},{"osd":3,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.376},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Sun Mar 24 04:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.436},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.31},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.34100000000000003,"15min":0.315},"min":{"1min":0.29799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Sun Mar 24 04:17:50 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.376,"15min":0.373},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.375}]},{"osd":3,"last update":"Sun Mar 24 04:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.316},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.35299999999999998}]},{"osd":4,"last update":"Sun Mar 24 04:17:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70299999999999996}]},{"osd":5,"last update":"Sun Mar 24 04:17:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.503},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Mar 24 04:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.62},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 04:18:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.30599999999999999},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Mar 24 04:17:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.32400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sun Mar 24 04:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42199999999999999,"15min":0.42399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.442,"15min":0.44500000000000001},"min":{"1min":0.28799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42899999999999999}]},{"osd":4,"last update":"Sun Mar 24 04:17:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65200000000000002}]},{"osd":5,"last update":"Sun Mar 24 04:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.81000000000000005}]},{"osd":6,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6232,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738920,"statfs":{"total":95995035648,"available":95988654080,"internally_reserved":0,"allocated":606208,"data_stored":244572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:17:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.371,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Mar 24 04:17:48 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38700000000000001,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.38400000000000001,"15min":0.36899999999999999},"min":{"1min":0.28299999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27200000000000002}]},{"osd":3,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.312},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.26300000000000001}]},{"osd":4,"last update":"Sun Mar 24 04:17:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55600000000000005}]},{"osd":5,"last update":"Sun Mar 24 04:18:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Mar 24 04:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5716,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739436,"statfs":{"total":95995035648,"available":95989182464,"internally_reserved":0,"allocated":602112,"data_stored":244645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42299999999999999,"15min":0.42799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.28799999999999998}]},{"osd":1,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.376},{"interface":"front","average":{"1min":0.438,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.308}]},{"osd":2,"last update":"Sun Mar 24 04:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48999999999999999,"15min":0.496},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.35699999999999998}]},{"osd":4,"last update":"Sun Mar 24 04:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86799999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Sun Mar 24 04:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.51700000000000002}]},{"osd":6,"last update":"Sun Mar 24 04:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.65200000000000002}]},{"osd":2,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67200000000000004}]},{"osd":3,"last update":"Sun Mar 24 04:17:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Mar 24 04:17:59 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.47399999999999998}]},{"osd":6,"last update":"Sun Mar 24 04:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.621},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57499999999999996}]},{"osd":1,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625}]},{"osd":4,"last update":"Sun Mar 24 04:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.54600000000000004}]},{"osd":6,"last update":"Sun Mar 24 04:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T04:18:43.692 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T04:18:43.692 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T04:18:43.692 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T04:18:43.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T04:18:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:43 smithi116 ceph-mon[27165]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:18:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:18:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:18:43.915 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T04:18:44.672 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:44 smithi022 ceph-mon[24756]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:44.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:44 smithi116 ceph-mon[27165]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:45.694 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:18:45.694 INFO:teuthology.orchestra.run.smithi022.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T04:18:46.214 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T04:18:46.214 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T04:18:46.214 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T04:18:46.224 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:18:46.225 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T04:18:46.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:46 smithi022 ceph-mon[24756]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:46.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:46 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1686916558' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T04:18:46.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:46 smithi116 ceph-mon[27165]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:46.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:46 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1686916558' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T04:18:48.123 INFO:teuthology.orchestra.run.smithi022.stdout:Backend: cephadm 2024-03-24T04:18:48.123 INFO:teuthology.orchestra.run.smithi022.stdout:Available: Yes 2024-03-24T04:18:48.123 INFO:teuthology.orchestra.run.smithi022.stdout:Paused: No 2024-03-24T04:18:48.636 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:48 smithi022 ceph-mon[24756]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:48.637 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:48 smithi022 ceph-mon[24756]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:48.642 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:48 smithi116 ceph-mon[27165]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:48.642 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:48 smithi116 ceph-mon[27165]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:48.867 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T04:18:49.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:18:49.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:18:49.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:18:49.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:49 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:18:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:18:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:18:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:18:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:49 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:18:50.838 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 4f0a417b24c7 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (4m) 65s ago 4m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (3m) 2m ago 5m 44.5M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (4m) 65s ago 4m 412M - 17.2.0 e1d6a67b021e 202afd7d8f70 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (6m) 2m ago 7m 41.7M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (4m) 65s ago 4m 40.7M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (5m) 2m ago 5m 16.9M - 1dbe0e931976 4f12550d5aec 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (4m) 65s ago 4m 15.8M - 1dbe0e931976 3ca47f34ef78 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (4m) 2m ago 4m 34.0M 3217M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (3m) 2m ago 3m 34.1M 3217M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (3m) 2m ago 3m 33.7M 3217M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:18:50.839 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (2m) 2m ago 2m 29.0M 3217M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:18:50.840 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (2m) 65s ago 2m 35.2M 3984M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:18:50.840 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (118s) 65s ago 117s 34.0M 3984M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:18:50.840 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (94s) 65s ago 94s 30.9M 3984M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:18:50.840 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (70s) 65s ago 70s 10.4M 3984M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:18:50.840 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (4m) 2m ago 4m 51.5M - 514e6a882f6e b3201eedfe34 2024-03-24T04:18:51.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:50 smithi022 ceph-mon[24756]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:51.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:50 smithi116 ceph-mon[27165]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:51.449 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T04:18:52.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:51 smithi022 ceph-mon[24756]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:52.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:51 smithi116 ceph-mon[27165]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:crash 2/2 2m ago 6m * 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:mgr 2/2 2m ago 6m count:2 2024-03-24T04:18:53.213 INFO:teuthology.orchestra.run.smithi022.stdout:mon 2/2 2m ago 5m smithi022:172.21.15.22=smithi022;smithi116:172.21.15.116=smithi116;count:2 2024-03-24T04:18:53.214 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-24T04:18:53.214 INFO:teuthology.orchestra.run.smithi022.stdout:osd 8 2m ago - 2024-03-24T04:18:53.214 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-24T04:18:53.214 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:52 smithi022 ceph-mon[24756]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:53.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:52 smithi116 ceph-mon[27165]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:53.828 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T04:18:55.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:54 smithi022 ceph-mon[24756]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:55.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:54 smithi022 ceph-mon[24756]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:55.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:54 smithi116 ceph-mon[27165]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:55.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:54 smithi116 ceph-mon[27165]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:55.742 INFO:teuthology.orchestra.run.smithi022.stdout:HOST ADDR LABELS STATUS 2024-03-24T04:18:55.742 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 172.21.15.22 2024-03-24T04:18:55.743 INFO:teuthology.orchestra.run.smithi022.stdout:smithi116 172.21.15.116 2024-03-24T04:18:55.743 INFO:teuthology.orchestra.run.smithi022.stdout:2 hosts in cluster 2024-03-24T04:18:56.443 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T04:18:57.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:56 smithi022 ceph-mon[24756]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:57.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:56 smithi022 ceph-mon[24756]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:56 smithi116 ceph-mon[27165]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:56 smithi116 ceph-mon[27165]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:18:58.299 INFO:teuthology.orchestra.run.smithi022.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-03-24T04:18:58.300 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402EC480DGN 480G LVM detected, locked 2024-03-24T04:18:58.300 INFO:teuthology.orchestra.run.smithi022.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 400G LVM detected, locked 2024-03-24T04:18:58.872 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T04:18:58.883 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:18:58.883 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-24T04:18:58.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ systemctl stop nfs-server 2024-03-24T04:18:58.919 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T04:18:58.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-24T04:18:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:+ systemctl stop nfs-server 2024-03-24T04:18:58.960 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T04:18:58.969 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:18:58.969 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-03-24T04:18:59.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:58 smithi116 ceph-mon[27165]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:18:59.276 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:58 smithi022 ceph-mon[24756]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:00.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:18:59 smithi022 ceph-mon[24756]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:18:59 smithi116 ceph-mon[27165]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:01.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:01 smithi116 ceph-mon[27165]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:01 smithi022 ceph-mon[24756]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-24T04:19:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-24T04:19:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T04:19:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-24T04:19:02.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:02 smithi116 ceph-mon[27165]: pgmap v207: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:02.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-mon[24756]: pgmap v207: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:02 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[24752]: 2024-03-24T04:19:02.268+0000 7fd560876700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:19:02.950 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-03-24T04:19:02.960 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-03-24T04:19:02.960 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: fsmap foofs:0 2024-03-24T04:19:03.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: Saving service mds.foofs spec with placement count:2 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T04:19:03.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:03.475 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:03 smithi022 ceph-mon[24756]: Deploying daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:19:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-24T04:19:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: fsmap foofs:0 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: Saving service mds.foofs spec with placement count:2 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:19:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T04:19:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:03.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:03 smithi116 ceph-mon[27165]: Deploying daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:19:04.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:04 smithi022 ceph-mon[24756]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T04:19:04.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:04 smithi022 ceph-mon[24756]: pgmap v211: 65 pgs: 2 creating+activating, 13 creating+peering, 6 active+clean, 44 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:04 smithi116 ceph-mon[27165]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T04:19:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:04 smithi116 ceph-mon[27165]: pgmap v211: 65 pgs: 2 creating+activating, 13 creating+peering, 6 active+clean, 44 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:05.334 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:05 smithi022 ceph-mon[24756]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T04:19:05.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:05 smithi116 ceph-mon[27165]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T04:19:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:19:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T04:12:37.186370Z", "last_refresh": "2024-03-24T04:16:21.656651Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T04:12:32.129867Z", "last_refresh": "2024-03-24T04:16:21.656719Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T04:12:34.573641Z", "last_refresh": "2024-03-24T04:16:21.656784Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T04:19:02.292980Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T04:19:02.287150Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T04:12:30.060499Z", "last_refresh": "2024-03-24T04:16:21.656577Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi022:172.21.15.22=smithi022", "smithi116:172.21.15.116=smithi116"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T04:13:20.654303Z", "last_refresh": "2024-03-24T04:16:21.656418Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T04:12:35.982152Z", "last_refresh": "2024-03-24T04:16:21.656861Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T04:16:21.656991Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T04:12:33.276375Z", "last_refresh": "2024-03-24T04:16:21.656928Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T04:19:06.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:06 smithi022 ceph-mon[24756]: pgmap v213: 65 pgs: 2 creating+activating, 13 creating+peering, 34 active+clean, 16 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:06.475 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:06 smithi022 ceph-mon[24756]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:06 smithi116 ceph-mon[27165]: pgmap v213: 65 pgs: 2 creating+activating, 13 creating+peering, 34 active+clean, 16 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:06 smithi116 ceph-mon[27165]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:06.873 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-24T04:19:07.874 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T04:19:08.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:08.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:19:08.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: Deploying daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: mds.? [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] up:boot 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: daemon mds.foofs.smithi022.ekdqmp assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: Cluster is now healthy 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: fsmap foofs:0 1 up:standby 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:19:08.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:creating} 2024-03-24T04:19:08.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: pgmap v214: 65 pgs: 2 creating+peering, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:08.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:08 smithi022 ceph-mon[24756]: daemon mds.foofs.smithi022.ekdqmp is now active in filesystem foofs as rank 0 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: Deploying daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: mds.? [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] up:boot 2024-03-24T04:19:08.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: daemon mds.foofs.smithi022.ekdqmp assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: Cluster is now healthy 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: fsmap foofs:0 1 up:standby 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:creating} 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: pgmap v214: 65 pgs: 2 creating+peering, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:08.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:08 smithi116 ceph-mon[27165]: daemon mds.foofs.smithi022.ekdqmp is now active in filesystem foofs as rank 0 2024-03-24T04:19:09.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:09 smithi022 ceph-mon[24756]: mds.? [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] up:active 2024-03-24T04:19:09.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:09 smithi022 ceph-mon[24756]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 2024-03-24T04:19:09.552 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:09 smithi116 ceph-mon[27165]: mds.? [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] up:active 2024-03-24T04:19:09.552 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:09 smithi116 ceph-mon[27165]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 2024-03-24T04:19:09.715 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:19:09.715 INFO:teuthology.orchestra.run.smithi022.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T04:12:37.186370Z", "last_refresh": "2024-03-24T04:16:21.656651Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T04:12:32.129867Z", "last_refresh": "2024-03-24T04:16:21.656719Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T04:12:34.573641Z", "last_refresh": "2024-03-24T04:16:21.656784Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T04:19:02.292980Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T04:19:02.287150Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T04:12:30.060499Z", "last_refresh": "2024-03-24T04:16:21.656577Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi022:172.21.15.22=smithi022", "smithi116:172.21.15.116=smithi116"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T04:13:20.654303Z", "last_refresh": "2024-03-24T04:16:21.656418Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T04:12:35.982152Z", "last_refresh": "2024-03-24T04:16:21.656861Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T04:16:21.656991Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T04:12:33.276375Z", "last_refresh": "2024-03-24T04:16:21.656928Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T04:19:10.355 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: pgmap v215: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 437 B/s wr, 1 op/s 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:10.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:10.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:boot 2024-03-24T04:19:10.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:19:10.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:19:10.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:10 smithi116 ceph-mon[27165]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: pgmap v215: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 437 B/s wr, 1 op/s 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:boot 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:19:11.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:10 smithi022 ceph-mon[24756]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:19:11.356 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T04:19:12.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:12 smithi116 ceph-mon[27165]: pgmap v216: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 637 B/s wr, 2 op/s 2024-03-24T04:19:12.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:12 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:13.019 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:12 smithi022 ceph-mon[24756]: pgmap v216: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 637 B/s wr, 2 op/s 2024-03-24T04:19:13.020 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:12 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:14.297 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:19:14.297 INFO:teuthology.orchestra.run.smithi022.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T04:12:37.186370Z", "last_refresh": "2024-03-24T04:16:21.656651Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T04:12:32.129867Z", "last_refresh": "2024-03-24T04:16:21.656719Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T04:12:34.573641Z", "last_refresh": "2024-03-24T04:16:21.656784Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T04:19:02.292980Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T04:19:02.287150Z", "last_refresh": "2024-03-24T04:19:11.620882Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T04:12:30.060499Z", "last_refresh": "2024-03-24T04:16:21.656577Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi022:172.21.15.22=smithi022", "smithi116:172.21.15.116=smithi116"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T04:13:20.654303Z", "last_refresh": "2024-03-24T04:16:21.656418Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T04:12:35.982152Z", "last_refresh": "2024-03-24T04:16:21.656861Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T04:16:21.656991Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T04:12:33.276375Z", "last_refresh": "2024-03-24T04:16:21.656928Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T04:19:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:14 smithi116 ceph-mon[27165]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T04:19:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:14 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:14.759 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:14 smithi022 ceph-mon[24756]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T04:19:14.759 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:14 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:15.061 INFO:tasks.cephadm:mds.foofs has 1/2 2024-03-24T04:19:15.710 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:15 smithi022 ceph-mon[24756]: from='client.14516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:15 smithi116 ceph-mon[27165]: from='client.14516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:16.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:16.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:16 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:16.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:16.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:16 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:18.083 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:19:18.083 INFO:teuthology.orchestra.run.smithi022.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T04:12:37.186370Z", "last_refresh": "2024-03-24T04:19:16.304381Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T04:12:32.129867Z", "last_refresh": "2024-03-24T04:19:11.620253Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-24T04:12:34.573641Z", "last_refresh": "2024-03-24T04:19:16.304515Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-24T04:19:02.292980Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T04:19:02.287150Z", "last_refresh": "2024-03-24T04:19:11.620882Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T04:12:30.060499Z", "last_refresh": "2024-03-24T04:19:11.620399Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi022:172.21.15.22=smithi022", "smithi116:172.21.15.116=smithi116"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T04:13:20.654303Z", "last_refresh": "2024-03-24T04:19:11.620473Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T04:12:35.982152Z", "last_refresh": "2024-03-24T04:19:11.620539Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-24T04:19:11.620603Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T04:12:33.276375Z", "last_refresh": "2024-03-24T04:19:16.304653Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T04:19:18.626 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:18 smithi022 ceph-mon[24756]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T04:19:18.627 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:18 smithi022 ceph-mon[24756]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:18.651 INFO:tasks.cephadm:mds.foofs has 2/2 2024-03-24T04:19:18.651 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T04:19:18.663 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:19:18.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-03-24T04:19:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:18 smithi116 ceph-mon[27165]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T04:19:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:18 smithi116 ceph-mon[27165]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:19:20.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:20 smithi022 ceph-mon[24756]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T04:19:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:20 smithi116 ceph-mon[27165]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T04:19:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:21 smithi116 ceph-mon[27165]: from='client.14524 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:21 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-24T04:19:21.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:21 smithi022 ceph-mon[24756]: from='client.14524 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:21.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:21 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-24T04:19:22.385 INFO:teuthology.orchestra.run.smithi022.stdout:NFS Cluster Created Successfully 2024-03-24T04:19:22.618 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:22 smithi022 ceph-mon[24756]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 853 B/s wr, 2 op/s 2024-03-24T04:19:22.618 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-24T04:19:22.618 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:22 smithi022 ceph-mon[24756]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T04:19:22.619 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:22 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-24T04:19:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:22 smithi116 ceph-mon[27165]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 853 B/s wr, 2 op/s 2024-03-24T04:19:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-24T04:19:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:22 smithi116 ceph-mon[27165]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T04:19:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:22 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-24T04:19:23.054 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-03-24T04:19:23.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-24T04:19:23.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T04:19:23.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Saving service nfs.foo spec with placement count:2 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T04:19:23.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Rados config object exists: conf-nfs.foo 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn-rgw 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T04:19:23.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.553 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:23 smithi022 ceph-mon[24756]: Deploying daemon nfs.foo.0.0.smithi022.iycgyn on smithi022 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Saving service nfs.foo spec with placement count:2 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Rados config object exists: conf-nfs.foo 2024-03-24T04:19:23.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn-rgw 2024-03-24T04:19:23.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:19:23.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T04:19:23.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:23.726 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:23 smithi116 ceph-mon[27165]: Deploying daemon nfs.foo.0.0.smithi022.iycgyn on smithi022 2024-03-24T04:19:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:24 smithi022 ceph-mon[24756]: pgmap v224: 97 pgs: 15 creating+peering, 14 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:24.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:24 smithi022 ceph-mon[24756]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T04:19:24.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:24 smithi022 ceph-mon[24756]: mgrmap e20: smithi022.ijpjvk(active, since 5m), standbys: smithi116.oxktjt 2024-03-24T04:19:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:24 smithi116 ceph-mon[27165]: pgmap v224: 97 pgs: 15 creating+peering, 14 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-24T04:19:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:24 smithi116 ceph-mon[27165]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T04:19:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:24 smithi116 ceph-mon[27165]: mgrmap e20: smithi022.ijpjvk(active, since 5m), standbys: smithi116.oxktjt 2024-03-24T04:19:26.119 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid command: Unexpected argument '--clusterid' 2024-03-24T04:19:26.119 INFO:teuthology.orchestra.run.smithi022.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-03-24T04:19:26.119 INFO:teuthology.orchestra.run.smithi022.stderr:Error EINVAL: invalid command 2024-03-24T04:19:26.473 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout: "bind": "/fake", 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout: "fs": "foofs", 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster": "foo", 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "RW" 2024-03-24T04:19:26.474 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:19:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:26 smithi116 ceph-mon[27165]: pgmap v226: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T04:19:26.741 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:26 smithi022 ceph-mon[24756]: pgmap v226: 97 pgs: 15 creating+peering, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T04:19:27.177 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T04:19:27.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T04:19:27.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:19:27.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:19:27.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:27 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:19:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:27 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:28.601 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:28 smithi022 ceph-mon[24756]: pgmap v227: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T04:19:28.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:28 smithi116 ceph-mon[27165]: pgmap v227: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T04:19:29.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:29 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T04:19:29.854 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:29 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:30.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:30 smithi022 ceph-mon[24756]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 128 B/s rd, 385 B/s wr, 0 op/s 2024-03-24T04:19:30.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:30 smithi022 ceph-mon[24756]: Rados config object exists: conf-nfs.foo 2024-03-24T04:19:30.798 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:30 smithi022 ceph-mon[24756]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt-rgw 2024-03-24T04:19:30.799 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:30 smithi022 ceph-mon[24756]: Deploying daemon nfs.foo.1.0.smithi116.wxmbpt on smithi116 2024-03-24T04:19:30.849 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:30 smithi116 ceph-mon[27165]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 128 B/s rd, 385 B/s wr, 0 op/s 2024-03-24T04:19:30.849 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:30 smithi116 ceph-mon[27165]: Rados config object exists: conf-nfs.foo 2024-03-24T04:19:30.849 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:30 smithi116 ceph-mon[27165]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt-rgw 2024-03-24T04:19:30.849 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:30 smithi116 ceph-mon[27165]: Deploying daemon nfs.foo.1.0.smithi116.wxmbpt on smithi116 2024-03-24T04:19:31.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:31 smithi022 ceph-mon[24756]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:31.804 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:31 smithi116 ceph-mon[27165]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:33.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:32 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:33.141 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T04:19:33.141 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:33.141 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:33.141 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:33.141 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:33.142 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:32 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:34.076 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:33 smithi116 ceph-mon[27165]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:34.076 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:33 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:33 smithi022 ceph-mon[24756]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:34.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:33 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:35.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:34 smithi022 ceph-mon[24756]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-24T04:19:35.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:34 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:35.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:34 smithi116 ceph-mon[27165]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-24T04:19:35.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:34 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:36.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:35 smithi022 ceph-mon[24756]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:36.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:35 smithi116 ceph-mon[27165]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:36.900 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:36 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-24T04:19:37.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:37.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:36 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:19:37.573 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T04:19:37.583 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:19:37.583 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-03-24T04:19:37.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ mkdir /mnt/foo 2024-03-24T04:19:37.618 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-03-24T04:19:37.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ hostname 2024-03-24T04:19:37.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ mount -t nfs smithi022:/fake /mnt/foo -o sync 2024-03-24T04:19:38.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:37 smithi022 ceph-mon[24756]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:38.099 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-03-24T04:19:38.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo test 2024-03-24T04:19:38.209 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-03-24T04:19:38.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:37 smithi116 ceph-mon[27165]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:19:38.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sync 2024-03-24T04:19:38.530 INFO:teuthology.run_tasks:Running task parallel... 2024-03-24T04:19:38.540 INFO:teuthology.task.parallel:starting parallel... 2024-03-24T04:19:38.540 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T04:19:38.541 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T04:19:38.541 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:19:38.541 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T04:19:38.542 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T04:19:38.542 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-24T04:19:38.543 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T04:19:38.543 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:19:38.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-03-24T04:19:38.684 INFO:teuthology.orchestra.run.smithi022.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T04:19:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:19:38.684 INFO:teuthology.orchestra.run.smithi022.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-24T04:19:38.685 INFO:teuthology.orchestra.run.smithi022.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-24T04:19:38.785 INFO:teuthology.orchestra.run.smithi022.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-24T04:19:38.785 INFO:teuthology.orchestra.run.smithi022.stdout:releasing clients 2024-03-24T04:19:39.045 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:38 smithi022 ceph-mon[24756]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-24T04:19:39.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:38 smithi116 ceph-mon[27165]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-24T04:19:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: 5 120 27.51 MB/sec warmup 1 sec latency 31.239 ms 2024-03-24T04:19:40.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 278 25.92 MB/sec warmup 2 sec latency 25.112 ms 2024-03-24T04:19:40.965 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:40 smithi022 ceph-mon[24756]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-24T04:19:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:40 smithi116 ceph-mon[27165]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-24T04:19:41.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 478 27.83 MB/sec warmup 3 sec latency 30.233 ms 2024-03-24T04:19:42.342 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T04:19:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 691 26.50 MB/sec warmup 4 sec latency 42.733 ms 2024-03-24T04:19:43.073 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:42 smithi022 ceph-mon[24756]: pgmap v234: 97 pgs: 97 active+clean; 24 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 41 op/s 2024-03-24T04:19:43.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:42 smithi116 ceph-mon[27165]: pgmap v234: 97 pgs: 97 active+clean; 24 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 41 op/s 2024-03-24T04:19:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 1137 22.99 MB/sec warmup 5 sec latency 22.975 ms 2024-03-24T04:19:44.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-24T04:19:44.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-24T04:19:44.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-24T04:19:44.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-24T04:19:44.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:19:44.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:19:44.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:19:44.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 2058 22.18 MB/sec warmup 6 sec latency 21.473 ms 2024-03-24T04:19:45.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: pgmap v235: 97 pgs: 97 active+clean; 62 MiB data, 87 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 5.2 MiB/s wr, 222 op/s 2024-03-24T04:19:45.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-24T04:19:45.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-24T04:19:45.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-24T04:19:45.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-24T04:19:45.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:44 smithi022 ceph-mon[24756]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: pgmap v235: 97 pgs: 97 active+clean; 62 MiB data, 87 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 5.2 MiB/s wr, 222 op/s 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-24T04:19:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:44 smithi116 ceph-mon[27165]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T04:19:45.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 2891 23.21 MB/sec warmup 7 sec latency 26.702 ms 2024-03-24T04:19:45.862 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:45 smithi022 ceph-mon[24756]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T04:19:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:45 smithi116 ceph-mon[27165]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T04:19:46.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T04:19:46.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 3726 24.35 MB/sec warmup 8 sec latency 24.722 ms 2024-03-24T04:19:46.871 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:46 smithi022 ceph-mon[24756]: pgmap v238: 97 pgs: 97 active+clean; 96 MiB data, 139 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 13 MiB/s wr, 608 op/s 2024-03-24T04:19:47.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:46 smithi116 ceph-mon[27165]: pgmap v238: 97 pgs: 97 active+clean; 96 MiB data, 139 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 13 MiB/s wr, 608 op/s 2024-03-24T04:19:47.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 4056 22.28 MB/sec warmup 9 sec latency 24.793 ms 2024-03-24T04:19:48.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:48 smithi022 ceph-mon[24756]: pgmap v239: 97 pgs: 97 active+clean; 125 MiB data, 212 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 995 op/s 2024-03-24T04:19:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:48 smithi116 ceph-mon[27165]: pgmap v239: 97 pgs: 97 active+clean; 125 MiB data, 212 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 995 op/s 2024-03-24T04:19:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 4863 21.40 MB/sec warmup 10 sec latency 54.022 ms 2024-03-24T04:19:49.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 5819 21.00 MB/sec warmup 11 sec latency 18.385 ms 2024-03-24T04:19:50.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:50 smithi022 ceph-mon[24756]: pgmap v240: 97 pgs: 97 active+clean; 127 MiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 15 MiB/s wr, 965 op/s 2024-03-24T04:19:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:50 smithi116 ceph-mon[27165]: pgmap v240: 97 pgs: 97 active+clean; 127 MiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 15 MiB/s wr, 965 op/s 2024-03-24T04:19:50.786 INFO:teuthology.orchestra.run.smithi022.stdout: 5 6522 21.75 MB/sec warmup 12 sec latency 35.436 ms 2024-03-24T04:19:51.469 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph mgr module enable nfs --force' 2024-03-24T04:19:51.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 7124 21.80 MB/sec warmup 13 sec latency 51.450 ms 2024-03-24T04:19:52.465 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:52 smithi022 ceph-mon[24756]: pgmap v241: 97 pgs: 97 active+clean; 132 MiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 13 MiB/s wr, 872 op/s 2024-03-24T04:19:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:52 smithi116 ceph-mon[27165]: pgmap v241: 97 pgs: 97 active+clean; 132 MiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 13 MiB/s wr, 872 op/s 2024-03-24T04:19:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 7611 21.18 MB/sec warmup 14 sec latency 31.902 ms 2024-03-24T04:19:53.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 8079 20.06 MB/sec warmup 15 sec latency 51.419 ms 2024-03-24T04:19:54.372 INFO:teuthology.orchestra.run.smithi022.stderr:module 'nfs' is already enabled 2024-03-24T04:19:54.478 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:54 smithi022 ceph-mon[24756]: pgmap v242: 97 pgs: 97 active+clean; 135 MiB data, 286 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 13 MiB/s wr, 921 op/s 2024-03-24T04:19:54.478 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:54 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2371567197' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-24T04:19:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:54 smithi116 ceph-mon[27165]: pgmap v242: 97 pgs: 97 active+clean; 135 MiB data, 286 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 13 MiB/s wr, 921 op/s 2024-03-24T04:19:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:54 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2371567197' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-24T04:19:54.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 8558 19.57 MB/sec warmup 16 sec latency 34.765 ms 2024-03-24T04:19:55.490 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:55 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2371567197' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-24T04:19:55.490 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:55 smithi022 ceph-mon[24756]: mgrmap e21: smithi022.ijpjvk(active, since 6m), standbys: smithi116.oxktjt 2024-03-24T04:19:55.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:55 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2371567197' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-24T04:19:55.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:55 smithi116 ceph-mon[27165]: mgrmap e21: smithi022.ijpjvk(active, since 6m), standbys: smithi116.oxktjt 2024-03-24T04:19:55.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 9312 19.23 MB/sec warmup 17 sec latency 29.565 ms 2024-03-24T04:19:56.137 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-24T04:19:56.385 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:56 smithi022 ceph-mon[24756]: pgmap v243: 97 pgs: 97 active+clean; 138 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 MiB/s wr, 772 op/s 2024-03-24T04:19:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:56 smithi116 ceph-mon[27165]: pgmap v243: 97 pgs: 97 active+clean; 138 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 MiB/s wr, 772 op/s 2024-03-24T04:19:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 9767 18.94 MB/sec warmup 18 sec latency 33.809 ms 2024-03-24T04:19:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 10240 19.18 MB/sec warmup 19 sec latency 75.347 ms 2024-03-24T04:19:58.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:19:58 smithi022 ceph-mon[24756]: pgmap v244: 97 pgs: 97 active+clean; 143 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 910 op/s 2024-03-24T04:19:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:19:58 smithi116 ceph-mon[27165]: pgmap v244: 97 pgs: 97 active+clean; 143 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 910 op/s 2024-03-24T04:19:58.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 10895 19.39 MB/sec warmup 20 sec latency 31.356 ms 2024-03-24T04:19:59.194 INFO:teuthology.orchestra.run.smithi022.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:19:59.787 INFO:teuthology.orchestra.run.smithi022.stdout: 5 11394 18.77 MB/sec warmup 21 sec latency 22.222 ms 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: pgmap v245: 97 pgs: 97 active+clean; 144 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 661 op/s 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:20:00.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:00 smithi022 ceph-mon[24756]: overall HEALTH_OK 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:20:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:20:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: pgmap v245: 97 pgs: 97 active+clean; 144 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 661 op/s 2024-03-24T04:20:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T04:20:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:00 smithi116 ceph-mon[27165]: overall HEALTH_OK 2024-03-24T04:20:00.634 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T04:20:00.634 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:20:00.634 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-24T04:20:00.791 INFO:teuthology.orchestra.run.smithi022.stdout: 5 12019 18.55 MB/sec warmup 22 sec latency 18.699 ms 2024-03-24T04:20:01.791 INFO:teuthology.orchestra.run.smithi022.stdout: 5 12686 18.32 MB/sec warmup 23 sec latency 29.246 ms 2024-03-24T04:20:02.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:02 smithi022 ceph-mon[24756]: pgmap v246: 97 pgs: 97 active+clean; 151 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 788 op/s 2024-03-24T04:20:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:02 smithi116 ceph-mon[27165]: pgmap v246: 97 pgs: 97 active+clean; 151 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 788 op/s 2024-03-24T04:20:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: 5 13195 18.02 MB/sec warmup 24 sec latency 46.187 ms 2024-03-24T04:20:03.791 INFO:teuthology.orchestra.run.smithi022.stdout: 5 13635 18.11 MB/sec warmup 25 sec latency 30.710 ms 2024-03-24T04:20:03.882 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:20:04.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:04 smithi022 ceph-mon[24756]: pgmap v247: 97 pgs: 97 active+clean; 156 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.6 MiB/s wr, 784 op/s 2024-03-24T04:20:04.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:04 smithi022 ceph-mon[24756]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:04 smithi116 ceph-mon[27165]: pgmap v247: 97 pgs: 97 active+clean; 156 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.6 MiB/s wr, 784 op/s 2024-03-24T04:20:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:04 smithi116 ceph-mon[27165]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:04.790 INFO:teuthology.orchestra.run.smithi022.stdout: 5 14112 18.08 MB/sec warmup 26 sec latency 32.260 ms 2024-03-24T04:20:04.853 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:20:04.853 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 28s ago 6m 22.1M - ba2b418f427c 4f0a417b24c7 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (6m) 28s ago 6m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (6m) 30s ago 6m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (5m) 28s ago 6m 54.3M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (58s) 28s ago 57s 14.7M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (55s) 30s ago 55s 13.4M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (8m) 28s ago 8m 460M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (6m) 30s ago 6m 413M - 17.2.0 e1d6a67b021e 202afd7d8f70 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (8m) 28s ago 8m 46.1M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (6m) 30s ago 6m 38.1M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (38s) 28s ago 38s 49.0M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (33s) 30s ago 33s 50.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (6m) 28s ago 6m 17.0M - 1dbe0e931976 4f12550d5aec 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (5m) 30s ago 5m 15.6M - 1dbe0e931976 3ca47f34ef78 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 28s ago 5m 43.2M 3217M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:20:04.854 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (4m) 28s ago 4m 42.8M 3217M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 28s ago 4m 41.8M 3217M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (3m) 28s ago 3m 43.2M 3217M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (3m) 30s ago 3m 39.5M 3984M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (3m) 30s ago 3m 41.0M 3984M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (2m) 30s ago 2m 39.5M 3984M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (2m) 30s ago 2m 40.6M 3984M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:20:04.855 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (5m) 28s ago 5m 62.3M - 514e6a882f6e b3201eedfe34 2024-03-24T04:20:05.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:05 smithi022 ceph-mon[24756]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:05.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:05 smithi022 ceph-mon[24756]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:05.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:05.488 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:20:05.488 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T04:20:05.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:20:05.488 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:20:05.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:05 smithi116 ceph-mon[27165]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:05.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:05 smithi116 ceph-mon[27165]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: 5 14547 17.98 MB/sec warmup 27 sec latency 34.881 ms 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "", 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 image" 2024-03-24T04:20:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:20:06.497 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:06 smithi022 ceph-mon[24756]: pgmap v248: 97 pgs: 97 active+clean; 158 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.2 MiB/s wr, 776 op/s 2024-03-24T04:20:06.497 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:06 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/418185677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:20:06.497 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:06 smithi022 ceph-mon[24756]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:06.668 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:20:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:06 smithi116 ceph-mon[27165]: pgmap v248: 97 pgs: 97 active+clean; 158 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.2 MiB/s wr, 776 op/s 2024-03-24T04:20:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:06 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/418185677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:20:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:06 smithi116 ceph-mon[27165]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:06.792 INFO:teuthology.orchestra.run.smithi022.stdout: 5 14980 17.50 MB/sec warmup 28 sec latency 28.850 ms 2024-03-24T04:20:07.507 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:07 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1856141784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:20:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:07 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1856141784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:20:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: 5 15676 17.39 MB/sec warmup 29 sec latency 21.494 ms 2024-03-24T04:20:08.512 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:08 smithi022 ceph-mon[24756]: pgmap v249: 97 pgs: 97 active+clean; 166 MiB data, 527 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 878 op/s 2024-03-24T04:20:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:08 smithi116 ceph-mon[27165]: pgmap v249: 97 pgs: 97 active+clean; 166 MiB data, 527 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 878 op/s 2024-03-24T04:20:08.792 INFO:teuthology.orchestra.run.smithi022.stdout: 5 16413 17.38 MB/sec warmup 30 sec latency 27.279 ms 2024-03-24T04:20:09.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 16852 17.13 MB/sec warmup 31 sec latency 32.469 ms 2024-03-24T04:20:10.541 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:10 smithi022 ceph-mon[24756]: pgmap v250: 97 pgs: 97 active+clean; 167 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.9 MiB/s wr, 630 op/s 2024-03-24T04:20:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:10 smithi116 ceph-mon[27165]: pgmap v250: 97 pgs: 97 active+clean; 167 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.9 MiB/s wr, 630 op/s 2024-03-24T04:20:10.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 17440 17.44 MB/sec warmup 32 sec latency 30.479 ms 2024-03-24T04:20:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 18047 17.54 MB/sec warmup 33 sec latency 39.330 ms 2024-03-24T04:20:12.560 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:12 smithi022 ceph-mon[24756]: pgmap v251: 97 pgs: 97 active+clean; 172 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8.9 MiB/s wr, 882 op/s 2024-03-24T04:20:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:12 smithi116 ceph-mon[27165]: pgmap v251: 97 pgs: 97 active+clean; 172 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8.9 MiB/s wr, 882 op/s 2024-03-24T04:20:12.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 18457 17.22 MB/sec warmup 34 sec latency 57.697 ms 2024-03-24T04:20:13.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 19006 16.96 MB/sec warmup 35 sec latency 21.780 ms 2024-03-24T04:20:14.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:14 smithi022 ceph-mon[24756]: pgmap v252: 97 pgs: 97 active+clean; 173 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 789 op/s 2024-03-24T04:20:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:14 smithi116 ceph-mon[27165]: pgmap v252: 97 pgs: 97 active+clean; 173 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 789 op/s 2024-03-24T04:20:14.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 19741 17.09 MB/sec warmup 36 sec latency 42.746 ms 2024-03-24T04:20:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 20578 17.16 MB/sec warmup 37 sec latency 53.770 ms 2024-03-24T04:20:16.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:16 smithi022 ceph-mon[24756]: pgmap v253: 97 pgs: 97 active+clean; 184 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.3 MiB/s wr, 874 op/s 2024-03-24T04:20:16.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:16 smithi116 ceph-mon[27165]: pgmap v253: 97 pgs: 97 active+clean; 184 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.3 MiB/s wr, 874 op/s 2024-03-24T04:20:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 21326 17.44 MB/sec warmup 38 sec latency 21.575 ms 2024-03-24T04:20:17.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 21792 17.43 MB/sec warmup 39 sec latency 53.089 ms 2024-03-24T04:20:18.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:18 smithi022 ceph-mon[24756]: pgmap v254: 97 pgs: 97 active+clean; 183 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 856 op/s 2024-03-24T04:20:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:18 smithi116 ceph-mon[27165]: pgmap v254: 97 pgs: 97 active+clean; 183 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 856 op/s 2024-03-24T04:20:18.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 22339 17.17 MB/sec warmup 40 sec latency 29.601 ms 2024-03-24T04:20:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 23080 17.18 MB/sec warmup 41 sec latency 19.383 ms 2024-03-24T04:20:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:20 smithi022 ceph-mon[24756]: pgmap v255: 97 pgs: 97 active+clean; 183 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-03-24T04:20:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:20 smithi116 ceph-mon[27165]: pgmap v255: 97 pgs: 97 active+clean; 183 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-03-24T04:20:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 23770 17.09 MB/sec warmup 42 sec latency 42.201 ms 2024-03-24T04:20:21.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 24401 17.27 MB/sec warmup 43 sec latency 26.497 ms 2024-03-24T04:20:22.446 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:22 smithi022 ceph-mon[24756]: pgmap v256: 97 pgs: 97 active+clean; 195 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 944 op/s 2024-03-24T04:20:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:22 smithi116 ceph-mon[27165]: pgmap v256: 97 pgs: 97 active+clean; 195 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 944 op/s 2024-03-24T04:20:22.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 25236 17.54 MB/sec warmup 44 sec latency 15.318 ms 2024-03-24T04:20:23.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 25763 17.35 MB/sec warmup 45 sec latency 36.631 ms 2024-03-24T04:20:24.461 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:24 smithi022 ceph-mon[24756]: pgmap v257: 97 pgs: 97 active+clean; 196 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 772 op/s 2024-03-24T04:20:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:24 smithi116 ceph-mon[27165]: pgmap v257: 97 pgs: 97 active+clean; 196 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 772 op/s 2024-03-24T04:20:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 26301 17.20 MB/sec warmup 46 sec latency 53.812 ms 2024-03-24T04:20:25.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 27049 17.22 MB/sec warmup 47 sec latency 33.820 ms 2024-03-24T04:20:26.506 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:26 smithi022 ceph-mon[24756]: pgmap v258: 97 pgs: 97 active+clean; 201 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 849 op/s 2024-03-24T04:20:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:26 smithi116 ceph-mon[27165]: pgmap v258: 97 pgs: 97 active+clean; 201 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 849 op/s 2024-03-24T04:20:26.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 27829 17.37 MB/sec warmup 48 sec latency 29.427 ms 2024-03-24T04:20:27.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 28700 17.68 MB/sec warmup 49 sec latency 22.067 ms 2024-03-24T04:20:28.519 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:28 smithi022 ceph-mon[24756]: pgmap v259: 97 pgs: 97 active+clean; 206 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 848 op/s 2024-03-24T04:20:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:28 smithi116 ceph-mon[27165]: pgmap v259: 97 pgs: 97 active+clean; 206 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 848 op/s 2024-03-24T04:20:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 29315 17.53 MB/sec warmup 50 sec latency 22.956 ms 2024-03-24T04:20:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 30008 17.48 MB/sec warmup 51 sec latency 28.728 ms 2024-03-24T04:20:30.528 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:30 smithi022 ceph-mon[24756]: pgmap v260: 97 pgs: 97 active+clean; 211 MiB data, 836 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 826 op/s 2024-03-24T04:20:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:30 smithi116 ceph-mon[27165]: pgmap v260: 97 pgs: 97 active+clean; 211 MiB data, 836 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 826 op/s 2024-03-24T04:20:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 30829 17.49 MB/sec warmup 52 sec latency 28.985 ms 2024-03-24T04:20:31.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 31449 17.55 MB/sec warmup 53 sec latency 29.408 ms 2024-03-24T04:20:32.538 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:32 smithi022 ceph-mon[24756]: pgmap v261: 97 pgs: 97 active+clean; 216 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-24T04:20:32.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:32 smithi116 ceph-mon[27165]: pgmap v261: 97 pgs: 97 active+clean; 216 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-24T04:20:32.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 32082 17.66 MB/sec warmup 54 sec latency 33.283 ms 2024-03-24T04:20:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 32586 17.63 MB/sec warmup 55 sec latency 28.176 ms 2024-03-24T04:20:34.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:34 smithi022 ceph-mon[24756]: pgmap v262: 97 pgs: 97 active+clean; 220 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 856 op/s 2024-03-24T04:20:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:34 smithi116 ceph-mon[27165]: pgmap v262: 97 pgs: 97 active+clean; 220 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 856 op/s 2024-03-24T04:20:34.793 INFO:teuthology.orchestra.run.smithi022.stdout: 5 33164 17.43 MB/sec warmup 56 sec latency 32.364 ms 2024-03-24T04:20:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 33826 17.45 MB/sec warmup 57 sec latency 61.476 ms 2024-03-24T04:20:36.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 34530 17.40 MB/sec warmup 58 sec latency 37.364 ms 2024-03-24T04:20:37.170 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:20:37.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:37 smithi022 ceph-mon[24756]: pgmap v263: 97 pgs: 97 active+clean; 227 MiB data, 985 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-03-24T04:20:37.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:37.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:20:37.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:37 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:37 smithi116 ceph-mon[27165]: pgmap v263: 97 pgs: 97 active+clean; 227 MiB data, 985 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-03-24T04:20:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:20:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:37 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:20:37.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 35277 17.58 MB/sec warmup 59 sec latency 21.541 ms 2024-03-24T04:20:38.174 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 61s ago 7m 22.1M - ba2b418f427c 4f0a417b24c7 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 61s ago 7m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (6m) 63s ago 6m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (5m) 61s ago 7m 54.3M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (92s) 61s ago 91s 14.7M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (88s) 63s ago 88s 13.4M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (8m) 61s ago 8m 460M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (6m) 63s ago 6m 413M - 17.2.0 e1d6a67b021e 202afd7d8f70 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (8m) 61s ago 8m 46.1M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (6m) 63s ago 6m 38.1M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:20:38.175 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (72s) 61s ago 71s 49.0M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (66s) 63s ago 66s 50.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (7m) 61s ago 7m 17.0M - 1dbe0e931976 4f12550d5aec 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (6m) 63s ago 6m 15.6M - 1dbe0e931976 3ca47f34ef78 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 61s ago 5m 43.2M 3217M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (5m) 61s ago 5m 42.8M 3217M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 61s ago 4m 41.8M 3217M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (4m) 61s ago 4m 43.2M 3217M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (4m) 63s ago 4m 39.5M 3984M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (3m) 63s ago 3m 41.0M 3984M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (3m) 63s ago 3m 39.5M 3984M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (2m) 63s ago 2m 40.6M 3984M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:20:38.176 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (6m) 61s ago 6m 62.3M - 514e6a882f6e b3201eedfe34 2024-03-24T04:20:38.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi022.ijpjvk) 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi116 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: pgmap v264: 97 pgs: 97 active+clean; 231 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:38 smithi022 ceph-mon[24756]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi022.ijpjvk) 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi116 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: pgmap v264: 97 pgs: 97 active+clean; 231 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:38 smithi116 ceph-mon[27165]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:20:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:20:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:20:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T04:20:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:20:38.692 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:20:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 35869 17.65 MB/sec warmup 60 sec latency 33.645 ms 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "0/23 daemons upgraded", 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 image on host smithi116" 2024-03-24T04:20:39.112 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:20:39.523 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:39 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2907638078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:20:39.523 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:39 smithi022 ceph-mon[24756]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:39.619 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:20:39.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:39 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2907638078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:20:39.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:39 smithi116 ceph-mon[27165]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:20:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 36238 17.48 MB/sec warmup 61 sec latency 33.230 ms 2024-03-24T04:20:40.529 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:40 smithi022 ceph-mon[24756]: pgmap v265: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-03-24T04:20:40.529 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:40 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1268246770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:20:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:40 smithi116 ceph-mon[27165]: pgmap v265: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-03-24T04:20:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:40 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1268246770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:20:40.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 36680 17.28 MB/sec warmup 62 sec latency 33.913 ms 2024-03-24T04:20:41.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 37452 17.34 MB/sec warmup 63 sec latency 21.280 ms 2024-03-24T04:20:42.544 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:42 smithi022 ceph-mon[24756]: pgmap v266: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 861 op/s 2024-03-24T04:20:42.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:42 smithi116 ceph-mon[27165]: pgmap v266: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 861 op/s 2024-03-24T04:20:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 38241 17.32 MB/sec warmup 64 sec latency 23.178 ms 2024-03-24T04:20:43.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:20:43.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:43 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:20:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:20:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:43 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:20:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 39134 17.59 MB/sec warmup 65 sec latency 18.688 ms 2024-03-24T04:20:44.562 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:44 smithi022 ceph-mon[24756]: pgmap v267: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 899 op/s 2024-03-24T04:20:44.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:44 smithi116 ceph-mon[27165]: pgmap v267: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 899 op/s 2024-03-24T04:20:44.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 39713 17.60 MB/sec warmup 66 sec latency 21.531 ms 2024-03-24T04:20:45.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 40328 17.44 MB/sec warmup 67 sec latency 16.701 ms 2024-03-24T04:20:46.575 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:46 smithi022 ceph-mon[24756]: pgmap v268: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.0 MiB/s wr, 892 op/s 2024-03-24T04:20:46.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:46 smithi116 ceph-mon[27165]: pgmap v268: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.0 MiB/s wr, 892 op/s 2024-03-24T04:20:46.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 41145 17.52 MB/sec warmup 68 sec latency 29.771 ms 2024-03-24T04:20:47.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 41949 17.59 MB/sec warmup 69 sec latency 20.586 ms 2024-03-24T04:20:48.333 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:48 smithi022 ceph-mon[24756]: pgmap v269: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.8 MiB/s wr, 882 op/s 2024-03-24T04:20:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:48 smithi116 ceph-mon[27165]: pgmap v269: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.8 MiB/s wr, 882 op/s 2024-03-24T04:20:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 42630 17.68 MB/sec warmup 70 sec latency 21.733 ms 2024-03-24T04:20:49.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 43180 17.69 MB/sec warmup 71 sec latency 36.518 ms 2024-03-24T04:20:50.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:50 smithi022 ceph-mon[24756]: pgmap v270: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 717 op/s 2024-03-24T04:20:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:50 smithi116 ceph-mon[27165]: pgmap v270: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 717 op/s 2024-03-24T04:20:50.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 43828 17.54 MB/sec warmup 72 sec latency 33.148 ms 2024-03-24T04:20:51.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 44607 17.60 MB/sec warmup 73 sec latency 27.162 ms 2024-03-24T04:20:52.565 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:52 smithi022 ceph-mon[24756]: pgmap v271: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 10 MiB/s wr, 919 op/s 2024-03-24T04:20:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:52 smithi116 ceph-mon[27165]: pgmap v271: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 10 MiB/s wr, 919 op/s 2024-03-24T04:20:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 45241 17.55 MB/sec warmup 74 sec latency 28.835 ms 2024-03-24T04:20:53.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 45805 17.59 MB/sec warmup 75 sec latency 35.092 ms 2024-03-24T04:20:54.579 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:54 smithi022 ceph-mon[24756]: pgmap v272: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 999 op/s 2024-03-24T04:20:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:54 smithi116 ceph-mon[27165]: pgmap v272: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 999 op/s 2024-03-24T04:20:54.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 46734 17.79 MB/sec warmup 76 sec latency 13.133 ms 2024-03-24T04:20:55.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 47395 17.65 MB/sec warmup 77 sec latency 22.703 ms 2024-03-24T04:20:56.590 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:56 smithi022 ceph-mon[24756]: pgmap v273: 97 pgs: 97 active+clean; 268 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 877 op/s 2024-03-24T04:20:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:56 smithi116 ceph-mon[27165]: pgmap v273: 97 pgs: 97 active+clean; 268 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 877 op/s 2024-03-24T04:20:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 48023 17.68 MB/sec warmup 78 sec latency 61.235 ms 2024-03-24T04:20:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: 5 48816 17.65 MB/sec warmup 79 sec latency 37.680 ms 2024-03-24T04:20:58.354 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:20:58 smithi022 ceph-mon[24756]: pgmap v274: 97 pgs: 97 active+clean; 273 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 890 op/s 2024-03-24T04:20:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:20:58 smithi116 ceph-mon[27165]: pgmap v274: 97 pgs: 97 active+clean; 273 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 890 op/s 2024-03-24T04:20:58.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 49628 17.81 MB/sec warmup 80 sec latency 25.563 ms 2024-03-24T04:20:59.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 50261 17.86 MB/sec warmup 81 sec latency 21.598 ms 2024-03-24T04:21:00.616 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:00 smithi022 ceph-mon[24756]: pgmap v275: 97 pgs: 97 active+clean; 272 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 719 op/s 2024-03-24T04:21:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:00 smithi116 ceph-mon[27165]: pgmap v275: 97 pgs: 97 active+clean; 272 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 719 op/s 2024-03-24T04:21:00.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 50862 17.73 MB/sec warmup 82 sec latency 26.788 ms 2024-03-24T04:21:01.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 51535 17.76 MB/sec warmup 83 sec latency 22.965 ms 2024-03-24T04:21:02.635 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:02 smithi022 ceph-mon[24756]: pgmap v276: 97 pgs: 97 active+clean; 279 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 952 op/s 2024-03-24T04:21:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:02 smithi116 ceph-mon[27165]: pgmap v276: 97 pgs: 97 active+clean; 279 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 952 op/s 2024-03-24T04:21:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 52238 17.72 MB/sec warmup 84 sec latency 27.986 ms 2024-03-24T04:21:03.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 52849 17.74 MB/sec warmup 85 sec latency 24.627 ms 2024-03-24T04:21:04.665 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:04 smithi022 ceph-mon[24756]: pgmap v277: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-03-24T04:21:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:04 smithi116 ceph-mon[27165]: pgmap v277: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-03-24T04:21:04.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 53503 17.82 MB/sec warmup 86 sec latency 25.724 ms 2024-03-24T04:21:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 54035 17.75 MB/sec warmup 87 sec latency 26.890 ms 2024-03-24T04:21:06.681 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:06 smithi022 ceph-mon[24756]: pgmap v278: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 856 op/s 2024-03-24T04:21:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:06 smithi116 ceph-mon[27165]: pgmap v278: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 856 op/s 2024-03-24T04:21:06.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 54699 17.70 MB/sec warmup 88 sec latency 39.848 ms 2024-03-24T04:21:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 55511 17.74 MB/sec warmup 89 sec latency 24.888 ms 2024-03-24T04:21:08.348 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:08 smithi022 ceph-mon[24756]: pgmap v279: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 926 op/s 2024-03-24T04:21:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:08 smithi116 ceph-mon[27165]: pgmap v279: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 926 op/s 2024-03-24T04:21:08.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 55999 17.64 MB/sec warmup 90 sec latency 38.651 ms 2024-03-24T04:21:09.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 56739 17.76 MB/sec warmup 91 sec latency 24.606 ms 2024-03-24T04:21:10.052 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:21:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:10 smithi116 ceph-mon[27165]: pgmap v280: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 777 op/s 2024-03-24T04:21:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:10 smithi116 ceph-mon[27165]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:10.634 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:10 smithi022 ceph-mon[24756]: pgmap v280: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 777 op/s 2024-03-24T04:21:10.635 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:10 smithi022 ceph-mon[24756]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:10.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 57232 17.77 MB/sec warmup 92 sec latency 41.559 ms 2024-03-24T04:21:10.906 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:21:10.906 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (6m) 94s ago 7m 22.1M - ba2b418f427c 4f0a417b24c7 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 94s ago 7m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (7m) 96s ago 7m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (6m) 94s ago 7m 54.3M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (2m) 94s ago 2m 14.7M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (2m) 96s ago 2m 13.4M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (9m) 94s ago 9m 460M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (7m) 96s ago 7m 413M - 17.2.0 e1d6a67b021e 202afd7d8f70 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (9m) 94s ago 9m 46.1M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (7m) 96s ago 7m 38.1M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (104s) 94s ago 104s 49.0M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (99s) 96s ago 99s 50.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (7m) 94s ago 7m 17.0M - 1dbe0e931976 4f12550d5aec 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (6m) 96s ago 6m 15.6M - 1dbe0e931976 3ca47f34ef78 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (6m) 94s ago 6m 43.2M 3217M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:21:10.907 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (5m) 94s ago 5m 42.8M 3217M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (5m) 94s ago 5m 41.8M 3217M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (5m) 94s ago 5m 43.2M 3217M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (4m) 96s ago 4m 39.5M 3984M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (4m) 96s ago 4m 41.0M 3984M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (3m) 96s ago 3m 39.5M 3984M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (3m) 96s ago 3m 40.6M 3984M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:21:10.908 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (6m) 94s ago 6m 62.3M - 514e6a882f6e b3201eedfe34 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:21:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:21:11.433 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:21:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 57795 17.66 MB/sec warmup 93 sec latency 46.124 ms 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "0/23 daemons upgraded", 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading mgr daemons" 2024-03-24T04:21:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: Upgrade: Updating mgr.smithi116.oxktjt 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:21:12.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:11 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2575151929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:21:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: Upgrade: Updating mgr.smithi116.oxktjt 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:21:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:11 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2575151929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:21:12.545 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:21:12.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 58480 17.67 MB/sec warmup 94 sec latency 19.736 ms 2024-03-24T04:21:13.168 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:12 smithi022 ceph-mon[24756]: pgmap v281: 97 pgs: 97 active+clean; 298 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-03-24T04:21:13.168 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:12 smithi022 ceph-mon[24756]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:13.168 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:12 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/3211715777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:21:13.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:12 smithi116 ceph-mon[27165]: pgmap v281: 97 pgs: 97 active+clean; 298 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-03-24T04:21:13.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:12 smithi116 ceph-mon[27165]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:13.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:12 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/3211715777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:21:13.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 59248 17.66 MB/sec warmup 95 sec latency 36.269 ms 2024-03-24T04:21:14.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 59783 17.63 MB/sec warmup 96 sec latency 32.500 ms 2024-03-24T04:21:15.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:15 smithi022 ceph-mon[24756]: pgmap v282: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 902 op/s 2024-03-24T04:21:15.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:15 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:15.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:15 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:15.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:15 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:15.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:15 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:21:15.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:15 smithi116 ceph-mon[27165]: pgmap v282: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 902 op/s 2024-03-24T04:21:15.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:15 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:15.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:15 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:15.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:15 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:15.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:15 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:21:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 60464 17.72 MB/sec warmup 97 sec latency 21.664 ms 2024-03-24T04:21:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: 5 61115 17.71 MB/sec warmup 98 sec latency 32.294 ms 2024-03-24T04:21:17.136 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:17 smithi116 ceph-mon[27165]: pgmap v283: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 846 op/s 2024-03-24T04:21:17.208 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:17 smithi022 ceph-mon[24756]: pgmap v283: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 846 op/s 2024-03-24T04:21:17.796 INFO:teuthology.orchestra.run.smithi022.stdout: 5 61682 17.63 MB/sec warmup 99 sec latency 46.031 ms 2024-03-24T04:21:18.471 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:18 smithi022 ceph-mon[24756]: pgmap v284: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.7 MiB/s wr, 897 op/s 2024-03-24T04:21:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:18 smithi116 ceph-mon[27165]: pgmap v284: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.7 MiB/s wr, 897 op/s 2024-03-24T04:21:18.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 62170 17.62 MB/sec warmup 100 sec latency 31.401 ms 2024-03-24T04:21:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 62678 17.54 MB/sec warmup 101 sec latency 36.046 ms 2024-03-24T04:21:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:20 smithi116 ceph-mon[27165]: pgmap v285: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 725 op/s 2024-03-24T04:21:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:20 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:20.499 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:20 smithi022 ceph-mon[24756]: pgmap v285: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 725 op/s 2024-03-24T04:21:20.499 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:20.499 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:20 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 63071 17.45 MB/sec warmup 102 sec latency 60.255 ms 2024-03-24T04:21:21.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 63564 17.45 MB/sec warmup 103 sec latency 43.899 ms 2024-03-24T04:21:22.434 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:22 smithi116 ceph-mon[27165]: pgmap v286: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 977 op/s 2024-03-24T04:21:22.538 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:22 smithi022 ceph-mon[24756]: pgmap v286: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 977 op/s 2024-03-24T04:21:22.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 64148 17.49 MB/sec warmup 104 sec latency 26.250 ms 2024-03-24T04:21:23.374 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:21:23.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:21:23.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:21:23.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:21:23.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:21:23.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:23 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:21:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:21:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:21:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:21:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:21:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:21:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:23 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:21:23.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 64559 17.43 MB/sec warmup 105 sec latency 90.956 ms 2024-03-24T04:21:24.380 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.3 MiB/s wr, 771 op/s 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: mgrmap e22: smithi022.ijpjvk(active, since 7m), standbys: smithi116.oxktjt 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr fail", "who": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:21:24.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:24 smithi022 ceph-mon[24756]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.3 MiB/s wr, 771 op/s 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: mgrmap e22: smithi022.ijpjvk(active, since 7m), standbys: smithi116.oxktjt 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:21:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr fail", "who": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:21:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:24 smithi116 ceph-mon[27165]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T04:21:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 64993 17.32 MB/sec warmup 106 sec latency 27.073 ms 2024-03-24T04:21:25.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi022.ijpjvk) 2024-03-24T04:21:25.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: Failing over to other MGR 2024-03-24T04:21:25.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:21:25.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "mgr fail", "who": "smithi022.ijpjvk"}]': finished 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: mgrmap e23: smithi116.oxktjt(active, starting, since 0.998955s) 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:21:25.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: Manager daemon smithi116.oxktjt is now available 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/trash_purge_schedule"}]: dispatch 2024-03-24T04:21:25.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:25 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/trash_purge_schedule"}]: dispatch 2024-03-24T04:21:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi022.ijpjvk) 2024-03-24T04:21:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: Failing over to other MGR 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.14204 172.21.15.22:0/3882148294' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "mgr fail", "who": "smithi022.ijpjvk"}]': finished 2024-03-24T04:21:25.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: mgrmap e23: smithi116.oxktjt(active, starting, since 0.998955s) 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: Manager daemon smithi116.oxktjt is now available 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/trash_purge_schedule"}]: dispatch 2024-03-24T04:21:25.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:25 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi116.oxktjt/trash_purge_schedule"}]: dispatch 2024-03-24T04:21:25.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 65477 17.28 MB/sec warmup 107 sec latency 29.003 ms 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: Deploying cephadm binary to smithi116 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: Deploying cephadm binary to smithi022 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: mgrmap e24: smithi116.oxktjt(active, since 2s) 2024-03-24T04:21:26.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:26 smithi022 ceph-mon[24756]: pgmap v3: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:26.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 66067 17.27 MB/sec warmup 108 sec latency 53.385 ms 2024-03-24T04:21:26.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:26.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:26.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: Deploying cephadm binary to smithi116 2024-03-24T04:21:26.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: Deploying cephadm binary to smithi022 2024-03-24T04:21:26.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: mgrmap e24: smithi116.oxktjt(active, since 2s) 2024-03-24T04:21:26.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:26 smithi116 ceph-mon[27165]: pgmap v3: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:27.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 66520 17.18 MB/sec warmup 109 sec latency 53.939 ms 2024-03-24T04:21:28.516 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: [24/Mar/2024:04:21:26] ENGINE Bus STARTING 2024-03-24T04:21:28.516 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: [24/Mar/2024:04:21:26] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T04:21:28.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: [24/Mar/2024:04:21:26] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T04:21:28.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: [24/Mar/2024:04:21:26] ENGINE Bus STARTED 2024-03-24T04:21:28.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: pgmap v4: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:28.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:28 smithi022 ceph-mon[24756]: mgrmap e25: smithi116.oxktjt(active, since 3s) 2024-03-24T04:21:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: [24/Mar/2024:04:21:26] ENGINE Bus STARTING 2024-03-24T04:21:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: [24/Mar/2024:04:21:26] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T04:21:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: [24/Mar/2024:04:21:26] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T04:21:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: [24/Mar/2024:04:21:26] ENGINE Bus STARTED 2024-03-24T04:21:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: pgmap v4: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:28 smithi116 ceph-mon[27165]: mgrmap e25: smithi116.oxktjt(active, since 3s) 2024-03-24T04:21:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 67023 17.18 MB/sec warmup 110 sec latency 46.219 ms 2024-03-24T04:21:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 67507 17.20 MB/sec warmup 111 sec latency 44.258 ms 2024-03-24T04:21:30.546 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:30 smithi022 ceph-mon[24756]: pgmap v5: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:30.546 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:30 smithi022 ceph-mon[24756]: mgrmap e26: smithi116.oxktjt(active, since 5s) 2024-03-24T04:21:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:30 smithi116 ceph-mon[27165]: pgmap v5: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:30 smithi116 ceph-mon[27165]: mgrmap e26: smithi116.oxktjt(active, since 5s) 2024-03-24T04:21:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 67944 17.18 MB/sec warmup 112 sec latency 31.791 ms 2024-03-24T04:21:31.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 68312 17.06 MB/sec warmup 113 sec latency 52.488 ms 2024-03-24T04:21:32.600 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:32 smithi022 ceph-mon[24756]: pgmap v6: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:32.666 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:32 smithi116 ceph-mon[27165]: pgmap v6: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-03-24T04:21:32.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 68547 16.94 MB/sec warmup 114 sec latency 105.699 ms 2024-03-24T04:21:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 68846 16.87 MB/sec warmup 115 sec latency 62.040 ms 2024-03-24T04:21:34.401 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-03-24T04:21:34.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: Standby manager daemon smithi022.ijpjvk started 2024-03-24T04:21:34.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/crt"}]: dispatch 2024-03-24T04:21:34.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:21:34.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/key"}]: dispatch 2024-03-24T04:21:34.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:34 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:21:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-03-24T04:21:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: Standby manager daemon smithi022.ijpjvk started 2024-03-24T04:21:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/crt"}]: dispatch 2024-03-24T04:21:34.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:21:34.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/key"}]: dispatch 2024-03-24T04:21:34.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:34 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/2620578612' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:21:34.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 69217 16.83 MB/sec warmup 116 sec latency 59.443 ms 2024-03-24T04:21:35.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:35 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:21:35.414 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:35 smithi022 ceph-mon[24756]: mgrmap e27: smithi116.oxktjt(active, since 10s), standbys: smithi022.ijpjvk 2024-03-24T04:21:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:35 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:21:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:35 smithi116 ceph-mon[27165]: mgrmap e27: smithi116.oxktjt(active, since 10s), standbys: smithi022.ijpjvk 2024-03-24T04:21:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 69698 16.77 MB/sec warmup 117 sec latency 84.392 ms 2024-03-24T04:21:36.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:36 smithi022 ceph-mon[24756]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 4.0 MiB/s wr, 392 op/s 2024-03-24T04:21:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:36 smithi116 ceph-mon[27165]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 4.0 MiB/s wr, 392 op/s 2024-03-24T04:21:36.802 INFO:teuthology.orchestra.run.smithi022.stdout: 5 70242 16.72 MB/sec warmup 118 sec latency 32.596 ms 2024-03-24T04:21:37.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 70761 16.72 MB/sec warmup 119 sec latency 33.941 ms 2024-03-24T04:21:38.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:38 smithi022 ceph-mon[24756]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 5.4 MiB/s wr, 641 op/s 2024-03-24T04:21:38.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:38 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:38.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:38 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:38 smithi116 ceph-mon[27165]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 5.4 MiB/s wr, 641 op/s 2024-03-24T04:21:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:38 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:38 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 71933 8.71 MB/sec execute 1 sec latency 26.157 ms 2024-03-24T04:21:40.466 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 587 op/s 2024-03-24T04:21:40.466 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.466 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.466 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:40 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 587 op/s 2024-03-24T04:21:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:40.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:40 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:40.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 72444 11.23 MB/sec execute 2 sec latency 36.896 ms 2024-03-24T04:21:41.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 73055 12.38 MB/sec execute 3 sec latency 32.783 ms 2024-03-24T04:21:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 587 op/s 2024-03-24T04:21:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi022 to 1937M 2024-03-24T04:21:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:42 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.495 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.9 MiB/s wr, 587 op/s 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi022 to 1937M 2024-03-24T04:21:42.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:42 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 73535 11.55 MB/sec execute 4 sec latency 30.428 ms 2024-03-24T04:21:42.998 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:21:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: 5 74039 12.04 MB/sec execute 5 sec latency 25.871 ms 2024-03-24T04:21:44.028 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (7m) 4s ago 8m 25.8M - 0.23.0 ba2b418f427c 4f0a417b24c7 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (8m) 4s ago 8m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (7m) 5s ago 7m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (6m) 4s ago 8m 54.5M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (2m) 4s ago 2m 34.1M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (2m) 5s ago 2m 14.1M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (9m) 4s ago 9m 330M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (29s) 5s ago 7m 489M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6ddb3fd00fbd 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (9m) 4s ago 9m 49.8M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (7m) 5s ago 7m 41.0M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (2m) 4s ago 2m 247M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (2m) 5s ago 2m 75.6M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:21:44.029 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (8m) 4s ago 8m 21.7M - 1.3.1 1dbe0e931976 4f12550d5aec 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (7m) 5s ago 7m 19.7M - 1.3.1 1dbe0e931976 3ca47f34ef78 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (6m) 4s ago 6m 295M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (6m) 4s ago 6m 227M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (6m) 4s ago 6m 186M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (5m) 4s ago 5m 234M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (5m) 5s ago 5m 247M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (4m) 5s ago 4m 285M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (4m) 5s ago 4m 255M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (4m) 5s ago 4m 352M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:21:44.030 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (7m) 4s ago 7m 66.7M - 2.33.4 514e6a882f6e b3201eedfe34 2024-03-24T04:21:44.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:44.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.6 MiB/s wr, 915 op/s 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: Adjusting osd_memory_target on smithi116 to 2704M 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:21:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:21:44.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:44 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:21:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:21:44.726 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 8.6 MiB/s wr, 915 op/s 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T04:21:44.779 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: Adjusting osd_memory_target on smithi116 to 2704M 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:21:44.780 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:44 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:21:44.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 74506 12.72 MB/sec execute 6 sec latency 26.027 ms 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:21:45.357 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T04:21:45.358 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "", 2024-03-24T04:21:45.358 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:21:45.358 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1556804700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:21:45.789 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.790 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:45 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 75024 13.77 MB/sec execute 7 sec latency 30.280 ms 2024-03-24T04:21:45.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1556804700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:45.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:45 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:21:46.105 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:21:46.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:46 smithi022 ceph-mon[24756]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T04:21:46.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:46 smithi022 ceph-mon[24756]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T04:21:46.795 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:46 smithi022 ceph-mon[24756]: pgmap v13: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 622 op/s 2024-03-24T04:21:46.795 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:46 smithi022 ceph-mon[24756]: from='client.24465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:46.795 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:46 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/391139945' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:21:46.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 75561 13.11 MB/sec execute 8 sec latency 24.906 ms 2024-03-24T04:21:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:46 smithi116 ceph-mon[27165]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T04:21:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:46 smithi116 ceph-mon[27165]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T04:21:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:46 smithi116 ceph-mon[27165]: pgmap v13: 97 pgs: 97 active+clean; 344 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 622 op/s 2024-03-24T04:21:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:46 smithi116 ceph-mon[27165]: from='client.24465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:21:46.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:46 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/391139945' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:21:47.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 75979 12.99 MB/sec execute 9 sec latency 41.284 ms 2024-03-24T04:21:48.563 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:48 smithi022 ceph-mon[24756]: pgmap v14: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 911 op/s 2024-03-24T04:21:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 76449 12.84 MB/sec execute 10 sec latency 40.692 ms 2024-03-24T04:21:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:48 smithi116 ceph-mon[27165]: pgmap v14: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 911 op/s 2024-03-24T04:21:49.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 77011 12.78 MB/sec execute 11 sec latency 28.767 ms 2024-03-24T04:21:50.582 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:50 smithi022 ceph-mon[24756]: pgmap v15: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-03-24T04:21:50.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 77441 12.71 MB/sec execute 12 sec latency 46.438 ms 2024-03-24T04:21:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:50 smithi116 ceph-mon[27165]: pgmap v15: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-03-24T04:21:51.806 INFO:teuthology.orchestra.run.smithi022.stdout: 5 77894 12.99 MB/sec execute 13 sec latency 39.471 ms 2024-03-24T04:21:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 78417 13.53 MB/sec execute 14 sec latency 30.158 ms 2024-03-24T04:21:52.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:52 smithi022 ceph-mon[24756]: pgmap v16: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-03-24T04:21:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:52 smithi116 ceph-mon[27165]: pgmap v16: 97 pgs: 97 active+clean; 349 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-03-24T04:21:53.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 78834 13.13 MB/sec execute 15 sec latency 34.963 ms 2024-03-24T04:21:54.597 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:54 smithi022 ceph-mon[24756]: pgmap v17: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 908 op/s 2024-03-24T04:21:54.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 79293 12.75 MB/sec execute 16 sec latency 23.386 ms 2024-03-24T04:21:54.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:54 smithi116 ceph-mon[27165]: pgmap v17: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 908 op/s 2024-03-24T04:21:55.601 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:55 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:55.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 79911 12.97 MB/sec execute 17 sec latency 21.632 ms 2024-03-24T04:21:55.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:55 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:21:56.621 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:56 smithi022 ceph-mon[24756]: pgmap v18: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-03-24T04:21:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 80504 13.10 MB/sec execute 18 sec latency 41.733 ms 2024-03-24T04:21:56.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:56 smithi116 ceph-mon[27165]: pgmap v18: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-03-24T04:21:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 81000 13.04 MB/sec execute 19 sec latency 38.745 ms 2024-03-24T04:21:58.642 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:21:58 smithi022 ceph-mon[24756]: pgmap v19: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 883 op/s 2024-03-24T04:21:58.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 81496 13.28 MB/sec execute 20 sec latency 21.724 ms 2024-03-24T04:21:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:21:58 smithi116 ceph-mon[27165]: pgmap v19: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 883 op/s 2024-03-24T04:21:59.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 82005 13.61 MB/sec execute 21 sec latency 36.375 ms 2024-03-24T04:22:00.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:00 smithi022 ceph-mon[24756]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.9 MiB/s wr, 594 op/s 2024-03-24T04:22:00.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 82439 13.38 MB/sec execute 22 sec latency 39.031 ms 2024-03-24T04:22:00.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:00 smithi116 ceph-mon[27165]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.9 MiB/s wr, 594 op/s 2024-03-24T04:22:01.807 INFO:teuthology.orchestra.run.smithi022.stdout: 5 82802 13.01 MB/sec execute 23 sec latency 32.412 ms 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 608 op/s 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:02 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:22:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: 5 83198 12.95 MB/sec execute 24 sec latency 59.661 ms 2024-03-24T04:22:02.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 608 op/s 2024-03-24T04:22:02.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:02.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:02 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:22:03.751 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:03 smithi022 ceph-mon[24756]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T04:22:03.751 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:03 smithi022 ceph-mon[24756]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T04:22:03.752 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:03 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:22:03.752 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:03 smithi022 ceph-mon[24756]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T04:22:03.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 83822 13.01 MB/sec execute 25 sec latency 30.166 ms 2024-03-24T04:22:03.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:03 smithi116 ceph-mon[27165]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T04:22:03.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:03 smithi116 ceph-mon[27165]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T04:22:03.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:03 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T04:22:03.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:03 smithi116 ceph-mon[27165]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T04:22:04.755 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:04 smithi022 ceph-mon[24756]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 889 op/s 2024-03-24T04:22:04.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 84393 13.01 MB/sec execute 26 sec latency 30.611 ms 2024-03-24T04:22:04.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:04 smithi116 ceph-mon[27165]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 889 op/s 2024-03-24T04:22:05.764 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:05 smithi022 ceph-mon[24756]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-03-24T04:22:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 84953 13.18 MB/sec execute 27 sec latency 38.402 ms 2024-03-24T04:22:05.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:05 smithi116 ceph-mon[27165]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-03-24T04:22:06.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 85467 13.47 MB/sec execute 28 sec latency 35.253 ms 2024-03-24T04:22:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 85874 13.37 MB/sec execute 29 sec latency 38.497 ms 2024-03-24T04:22:08.545 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:08 smithi022 ceph-mon[24756]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 904 op/s 2024-03-24T04:22:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:08 smithi116 ceph-mon[27165]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 904 op/s 2024-03-24T04:22:08.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 86239 13.09 MB/sec execute 30 sec latency 33.127 ms 2024-03-24T04:22:09.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 86704 13.07 MB/sec execute 31 sec latency 32.667 ms 2024-03-24T04:22:10.570 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:10 smithi022 ceph-mon[24756]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.3 MiB/s wr, 600 op/s 2024-03-24T04:22:10.570 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:10 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:10 smithi116 ceph-mon[27165]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.3 MiB/s wr, 600 op/s 2024-03-24T04:22:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:10 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:10.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 87243 13.11 MB/sec execute 32 sec latency 29.051 ms 2024-03-24T04:22:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 87725 12.98 MB/sec execute 33 sec latency 31.348 ms 2024-03-24T04:22:12.586 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:12 smithi022 ceph-mon[24756]: pgmap v26: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.5 MiB/s wr, 609 op/s 2024-03-24T04:22:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:12 smithi116 ceph-mon[27165]: pgmap v26: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.5 MiB/s wr, 609 op/s 2024-03-24T04:22:12.811 INFO:teuthology.orchestra.run.smithi022.stdout: 5 88165 12.98 MB/sec execute 34 sec latency 40.190 ms 2024-03-24T04:22:13.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 88621 13.14 MB/sec execute 35 sec latency 25.748 ms 2024-03-24T04:22:14.616 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:14 smithi022 ceph-mon[24756]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 866 op/s 2024-03-24T04:22:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:14 smithi116 ceph-mon[27165]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 866 op/s 2024-03-24T04:22:14.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 89013 13.14 MB/sec execute 36 sec latency 32.057 ms 2024-03-24T04:22:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 89398 13.06 MB/sec execute 37 sec latency 34.199 ms 2024-03-24T04:22:16.576 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:22:16.614 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:16 smithi022 ceph-mon[24756]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-03-24T04:22:16.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:16 smithi116 ceph-mon[27165]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-03-24T04:22:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 89743 12.83 MB/sec execute 38 sec latency 32.743 ms 2024-03-24T04:22:17.386 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 starting - - - - 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (9m) 37s ago 9m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (8m) 39s ago 8m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (7m) 37s ago 8m 54.5M - 8.3.5 dad864ee21e9 cbd95639dea4 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (3m) 37s ago 3m 34.1M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (3m) 39s ago 3m 14.1M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (10m) 37s ago 10m 330M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (63s) 39s ago 8m 489M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6ddb3fd00fbd 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (10m) 37s ago 10m 49.8M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (8m) 39s ago 8m 41.0M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (2m) 37s ago 2m 247M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (2m) 39s ago 2m 75.6M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (8m) 37s ago 8m 21.7M - 1.3.1 1dbe0e931976 4f12550d5aec 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 39s ago 8m 19.7M - 1.3.1 1dbe0e931976 3ca47f34ef78 2024-03-24T04:22:17.387 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (7m) 37s ago 7m 295M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (7m) 37s ago 7m 227M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (6m) 37s ago 6m 186M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (6m) 37s ago 6m 234M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (5m) 39s ago 5m 247M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (5m) 39s ago 5m 285M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (5m) 39s ago 5m 255M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (4m) 39s ago 4m 352M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:22:17.388 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (7m) 37s ago 7m 66.7M - 2.33.4 514e6a882f6e b3201eedfe34 2024-03-24T04:22:17.630 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:17 smithi022 ceph-mon[24756]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:17.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:17 smithi116 ceph-mon[27165]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:17.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 90119 12.72 MB/sec execute 39 sec latency 41.579 ms 2024-03-24T04:22:18.044 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:22:18.044 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:22:18.044 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:22:18.045 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:22:18.385 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:18 smithi022 ceph-mon[24756]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:18.615 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:22:18.615 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:22:18.615 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:22:18.615 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:22:18.616 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:22:18.616 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T04:22:18.616 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "", 2024-03-24T04:22:18.616 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:22:18.616 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:22:18.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:18 smithi022 ceph-mon[24756]: pgmap v29: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 834 op/s 2024-03-24T04:22:18.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:18 smithi022 ceph-mon[24756]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:18.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:18 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2756323862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:22:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:18 smithi116 ceph-mon[27165]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:18 smithi116 ceph-mon[27165]: pgmap v29: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 834 op/s 2024-03-24T04:22:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:18 smithi116 ceph-mon[27165]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:18 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2756323862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:22:18.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 90604 12.81 MB/sec execute 40 sec latency 33.537 ms 2024-03-24T04:22:19.270 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:22:19.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:19 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/1414973195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:22:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:19 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1414973195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:22:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 91174 12.75 MB/sec execute 41 sec latency 39.931 ms 2024-03-24T04:22:20.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:20 smithi022 ceph-mon[24756]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:20.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:20 smithi022 ceph-mon[24756]: pgmap v30: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 528 op/s 2024-03-24T04:22:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:20 smithi116 ceph-mon[27165]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:20 smithi116 ceph-mon[27165]: pgmap v30: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 528 op/s 2024-03-24T04:22:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 91699 12.78 MB/sec execute 42 sec latency 27.815 ms 2024-03-24T04:22:21.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 92342 13.05 MB/sec execute 43 sec latency 22.980 ms 2024-03-24T04:22:22.422 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:22 smithi022 ceph-mon[24756]: pgmap v31: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 541 op/s 2024-03-24T04:22:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:22 smithi116 ceph-mon[27165]: pgmap v31: 97 pgs: 97 active+clean; 340 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 541 op/s 2024-03-24T04:22:22.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 92770 13.06 MB/sec execute 44 sec latency 39.996 ms 2024-03-24T04:22:23.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 93252 12.95 MB/sec execute 45 sec latency 30.557 ms 2024-03-24T04:22:24.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:24 smithi022 ceph-mon[24756]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 810 op/s 2024-03-24T04:22:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:24 smithi116 ceph-mon[27165]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 810 op/s 2024-03-24T04:22:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 93651 12.87 MB/sec execute 46 sec latency 33.683 ms 2024-03-24T04:22:25.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 94065 12.84 MB/sec execute 47 sec latency 52.759 ms 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: Reconfiguring node-exporter.smithi022 (dependencies changed)... 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: Reconfiguring daemon node-exporter.smithi022 on smithi022 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.6 MiB/s wr, 540 op/s 2024-03-24T04:22:26.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:26 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: Reconfiguring node-exporter.smithi022 (dependencies changed)... 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: Reconfiguring daemon node-exporter.smithi022 on smithi022 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.6 MiB/s wr, 540 op/s 2024-03-24T04:22:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:26 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:26.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 94661 12.87 MB/sec execute 48 sec latency 28.132 ms 2024-03-24T04:22:27.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 95022 12.71 MB/sec execute 49 sec latency 64.465 ms 2024-03-24T04:22:28.507 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:28 smithi022 ceph-mon[24756]: pgmap v34: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-03-24T04:22:28.508 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:28.508 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:28 smithi116 ceph-mon[27165]: pgmap v34: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-03-24T04:22:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 95505 12.80 MB/sec execute 50 sec latency 43.394 ms 2024-03-24T04:22:29.537 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:29 smithi022 ceph-mon[24756]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T04:22:29.537 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:29 smithi022 ceph-mon[24756]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T04:22:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:29 smithi116 ceph-mon[27165]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T04:22:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:29 smithi116 ceph-mon[27165]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T04:22:29.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 95918 12.89 MB/sec execute 51 sec latency 37.778 ms 2024-03-24T04:22:30.552 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:30 smithi022 ceph-mon[24756]: pgmap v35: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-03-24T04:22:30.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:30 smithi116 ceph-mon[27165]: pgmap v35: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.7 MiB/s wr, 580 op/s 2024-03-24T04:22:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 96277 12.87 MB/sec execute 52 sec latency 38.438 ms 2024-03-24T04:22:31.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 96673 12.78 MB/sec execute 53 sec latency 49.242 ms 2024-03-24T04:22:32.594 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:32 smithi022 ceph-mon[24756]: pgmap v36: 97 pgs: 97 active+clean; 343 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.2 MiB/s wr, 596 op/s 2024-03-24T04:22:32.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:32 smithi116 ceph-mon[27165]: pgmap v36: 97 pgs: 97 active+clean; 343 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.2 MiB/s wr, 596 op/s 2024-03-24T04:22:32.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 97039 12.67 MB/sec execute 54 sec latency 37.902 ms 2024-03-24T04:22:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 97476 12.68 MB/sec execute 55 sec latency 72.085 ms 2024-03-24T04:22:34.606 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:34 smithi022 ceph-mon[24756]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 818 op/s 2024-03-24T04:22:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:34 smithi116 ceph-mon[27165]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 818 op/s 2024-03-24T04:22:34.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 97970 12.64 MB/sec execute 56 sec latency 40.218 ms 2024-03-24T04:22:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 98352 12.54 MB/sec execute 57 sec latency 63.060 ms 2024-03-24T04:22:36.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:36 smithi022 ceph-mon[24756]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 540 op/s 2024-03-24T04:22:36.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:36 smithi116 ceph-mon[27165]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 540 op/s 2024-03-24T04:22:36.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 98776 12.53 MB/sec execute 58 sec latency 43.881 ms 2024-03-24T04:22:37.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 99103 12.51 MB/sec execute 59 sec latency 39.588 ms 2024-03-24T04:22:38.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:38 smithi022 ceph-mon[24756]: pgmap v39: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-03-24T04:22:38.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:38 smithi116 ceph-mon[27165]: pgmap v39: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-03-24T04:22:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: 5 99660 12.71 MB/sec execute 60 sec latency 39.621 ms 2024-03-24T04:22:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 100041 12.63 MB/sec execute 61 sec latency 44.420 ms 2024-03-24T04:22:40.460 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:40 smithi022 ceph-mon[24756]: pgmap v40: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-03-24T04:22:40.461 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:40 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:40 smithi116 ceph-mon[27165]: pgmap v40: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-03-24T04:22:40.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:40 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:40.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 100328 12.48 MB/sec execute 62 sec latency 48.346 ms 2024-03-24T04:22:41.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 100664 12.43 MB/sec execute 63 sec latency 44.623 ms 2024-03-24T04:22:42.632 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:42 smithi022 ceph-mon[24756]: pgmap v41: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.8 MiB/s wr, 533 op/s 2024-03-24T04:22:42.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:42 smithi116 ceph-mon[27165]: pgmap v41: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.8 MiB/s wr, 533 op/s 2024-03-24T04:22:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 100991 12.39 MB/sec execute 64 sec latency 120.349 ms 2024-03-24T04:22:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: 5 101336 12.33 MB/sec execute 65 sec latency 63.493 ms 2024-03-24T04:22:44.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:44 smithi022 ceph-mon[24756]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 728 op/s 2024-03-24T04:22:44.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:44 smithi116 ceph-mon[27165]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 728 op/s 2024-03-24T04:22:44.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 101831 12.27 MB/sec execute 66 sec latency 30.292 ms 2024-03-24T04:22:45.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 102286 12.25 MB/sec execute 67 sec latency 37.949 ms 2024-03-24T04:22:46.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:46 smithi022 ceph-mon[24756]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 492 op/s 2024-03-24T04:22:46.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:46 smithi116 ceph-mon[27165]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 492 op/s 2024-03-24T04:22:46.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 102694 12.29 MB/sec execute 68 sec latency 32.573 ms 2024-03-24T04:22:47.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 103153 12.39 MB/sec execute 69 sec latency 39.211 ms 2024-03-24T04:22:48.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:48 smithi022 ceph-mon[24756]: pgmap v44: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 729 op/s 2024-03-24T04:22:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:48 smithi116 ceph-mon[27165]: pgmap v44: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 729 op/s 2024-03-24T04:22:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 103579 12.37 MB/sec execute 70 sec latency 30.975 ms 2024-03-24T04:22:49.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 103944 12.26 MB/sec execute 71 sec latency 26.250 ms 2024-03-24T04:22:49.825 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:22:50.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:50 smithi022 ceph-mon[24756]: pgmap v45: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 473 op/s 2024-03-24T04:22:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:50 smithi116 ceph-mon[27165]: pgmap v45: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 473 op/s 2024-03-24T04:22:50.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 104354 12.23 MB/sec execute 72 sec latency 39.591 ms 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 starting - - - - 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (9m) 71s ago 9m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (8m) 72s ago 8m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 starting - - - - 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (3m) 71s ago 3m 34.1M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (3m) 72s ago 3m 14.1M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:9283 running (10m) 71s ago 10m 330M - 17.2.0 e1d6a67b021e fa59a8be63e0 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (96s) 72s ago 8m 489M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6ddb3fd00fbd 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (10m) 71s ago 11m 49.8M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (8m) 72s ago 8m 41.0M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (3m) 71s ago 3m 247M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:22:50.870 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (3m) 72s ago 3m 75.6M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 starting - - - - 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 72s ago 8m 19.7M - 1.3.1 1dbe0e931976 3ca47f34ef78 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (8m) 71s ago 8m 295M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (7m) 71s ago 7m 227M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (7m) 71s ago 7m 186M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (6m) 71s ago 6m 234M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (6m) 72s ago 6m 247M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (5m) 72s ago 5m 285M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (5m) 72s ago 5m 255M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (5m) 72s ago 5m 352M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:22:50.871 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (8m) 71s ago 8m 66.7M - 2.33.4 514e6a882f6e b3201eedfe34 2024-03-24T04:22:51.479 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:51 smithi022 ceph-mon[24756]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:51.480 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:51 smithi022 ceph-mon[24756]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:22:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:22:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:22:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T04:22:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:22:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:22:51.578 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:22:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:51 smithi116 ceph-mon[27165]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:51 smithi116 ceph-mon[27165]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:51.817 INFO:teuthology.orchestra.run.smithi022.stdout: 5 104758 12.24 MB/sec execute 73 sec latency 41.017 ms 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "", 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:22:52.146 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:22:52.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:52 smithi022 ceph-mon[24756]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:52.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:52 smithi022 ceph-mon[24756]: pgmap v46: 97 pgs: 97 active+clean; 334 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.4 MiB/s wr, 492 op/s 2024-03-24T04:22:52.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:52 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/781063200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:22:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:52 smithi116 ceph-mon[27165]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:52 smithi116 ceph-mon[27165]: pgmap v46: 97 pgs: 97 active+clean; 334 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.4 MiB/s wr, 492 op/s 2024-03-24T04:22:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:52 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/781063200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:22:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 105189 12.20 MB/sec execute 74 sec latency 52.869 ms 2024-03-24T04:22:52.820 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:22:53.494 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:53 smithi022 ceph-mon[24756]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:53.494 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:53 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2755785660' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:22:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:53 smithi116 ceph-mon[27165]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:22:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:53 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2755785660' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:22:53.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 105611 12.12 MB/sec execute 75 sec latency 29.535 ms 2024-03-24T04:22:54.512 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:54 smithi022 ceph-mon[24756]: pgmap v47: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.1 MiB/s wr, 704 op/s 2024-03-24T04:22:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:54 smithi116 ceph-mon[27165]: pgmap v47: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.1 MiB/s wr, 704 op/s 2024-03-24T04:22:54.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 106015 12.13 MB/sec execute 76 sec latency 31.482 ms 2024-03-24T04:22:55.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 106538 12.28 MB/sec execute 77 sec latency 33.058 ms 2024-03-24T04:22:56.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:56.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:56.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: Reconfiguring node-exporter.smithi116 (dependencies changed)... 2024-03-24T04:22:56.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: Reconfiguring daemon node-exporter.smithi116 on smithi116 2024-03-24T04:22:56.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: pgmap v48: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-03-24T04:22:56.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:55 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: Reconfiguring node-exporter.smithi116 (dependencies changed)... 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: Reconfiguring daemon node-exporter.smithi116 on smithi116 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: pgmap v48: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-03-24T04:22:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:55 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:22:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 106900 12.25 MB/sec execute 78 sec latency 43.886 ms 2024-03-24T04:22:57.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:56 smithi022 ceph-mon[24756]: mgrmap e28: smithi116.oxktjt(active, since 91s), standbys: smithi022.ijpjvk 2024-03-24T04:22:57.098 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:56 smithi116 ceph-mon[27165]: mgrmap e28: smithi116.oxktjt(active, since 91s), standbys: smithi022.ijpjvk 2024-03-24T04:22:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 107295 12.20 MB/sec execute 79 sec latency 42.313 ms 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: pgmap v49: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 744 op/s 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:22:58.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:22:58.344 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:22:58 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:22:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: pgmap v49: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 744 op/s 2024-03-24T04:22:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T04:22:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:22:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:22:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:22:58.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:22:58 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:22:58.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 107795 12.19 MB/sec execute 80 sec latency 22.180 ms 2024-03-24T04:22:59.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 108393 12.27 MB/sec execute 81 sec latency 32.206 ms 2024-03-24T04:23:00.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:00 smithi022 ceph-mon[24756]: pgmap v50: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 508 op/s 2024-03-24T04:23:00.524 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:00 smithi116 ceph-mon[27165]: pgmap v50: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 508 op/s 2024-03-24T04:23:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 109044 12.22 MB/sec execute 82 sec latency 41.177 ms 2024-03-24T04:23:01.371 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: Upgrade: Updating mgr.smithi022.ijpjvk 2024-03-24T04:23:01.371 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:01.371 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:01.371 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:01.371 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:23:01.372 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:01.372 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:01 smithi022 ceph-mon[24756]: Deploying daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:23:01.539 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: Upgrade: Updating mgr.smithi022.ijpjvk 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:01.540 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:01 smithi116 ceph-mon[27165]: Deploying daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:23:01.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 109530 12.26 MB/sec execute 83 sec latency 56.375 ms 2024-03-24T04:23:02.378 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:02 smithi022 ceph-mon[24756]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.7 MiB/s wr, 653 op/s 2024-03-24T04:23:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:02 smithi116 ceph-mon[27165]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.7 MiB/s wr, 653 op/s 2024-03-24T04:23:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 110146 12.42 MB/sec execute 84 sec latency 58.002 ms 2024-03-24T04:23:03.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 110609 12.43 MB/sec execute 85 sec latency 62.642 ms 2024-03-24T04:23:04.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:04 smithi022 ceph-mon[24756]: pgmap v52: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 801 op/s 2024-03-24T04:23:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:04 smithi116 ceph-mon[27165]: pgmap v52: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 801 op/s 2024-03-24T04:23:04.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 110881 12.33 MB/sec execute 86 sec latency 44.910 ms 2024-03-24T04:23:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 111194 12.27 MB/sec execute 87 sec latency 51.585 ms 2024-03-24T04:23:06.459 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:06 smithi022 ceph-mon[24756]: pgmap v53: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-03-24T04:23:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:06 smithi116 ceph-mon[27165]: pgmap v53: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-03-24T04:23:06.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 111490 12.19 MB/sec execute 88 sec latency 49.150 ms 2024-03-24T04:23:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 111815 12.19 MB/sec execute 89 sec latency 48.057 ms 2024-03-24T04:23:08.505 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:08 smithi022 ceph-mon[24756]: pgmap v54: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 817 op/s 2024-03-24T04:23:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:08 smithi116 ceph-mon[27165]: pgmap v54: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 817 op/s 2024-03-24T04:23:08.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 112178 12.11 MB/sec execute 90 sec latency 46.058 ms 2024-03-24T04:23:09.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 112545 12.05 MB/sec execute 91 sec latency 42.170 ms 2024-03-24T04:23:10.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: pgmap v55: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.9 MiB/s wr, 566 op/s 2024-03-24T04:23:10.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:10.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:10 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: pgmap v55: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.9 MiB/s wr, 566 op/s 2024-03-24T04:23:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:10 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:10.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 112882 12.01 MB/sec execute 92 sec latency 42.376 ms 2024-03-24T04:23:11.339 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:11 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:11 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 113249 12.03 MB/sec execute 93 sec latency 31.533 ms 2024-03-24T04:23:12.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:12 smithi022 ceph-mon[24756]: pgmap v56: 97 pgs: 97 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.5 MiB/s wr, 607 op/s 2024-03-24T04:23:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:12 smithi116 ceph-mon[27165]: pgmap v56: 97 pgs: 97 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.5 MiB/s wr, 607 op/s 2024-03-24T04:23:12.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 113654 12.07 MB/sec execute 94 sec latency 46.802 ms 2024-03-24T04:23:13.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 114018 12.07 MB/sec execute 95 sec latency 39.934 ms 2024-03-24T04:23:14.393 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:14 smithi022 ceph-mon[24756]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 633 op/s 2024-03-24T04:23:14.394 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:14 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:14.394 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:14 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:14 smithi116 ceph-mon[27165]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 633 op/s 2024-03-24T04:23:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:14 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:14 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:14.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 114289 12.02 MB/sec execute 96 sec latency 51.928 ms 2024-03-24T04:23:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 114584 11.97 MB/sec execute 97 sec latency 44.195 ms 2024-03-24T04:23:16.422 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:16 smithi022 ceph-mon[24756]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-03-24T04:23:16.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:16 smithi116 ceph-mon[27165]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-03-24T04:23:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 114976 11.92 MB/sec execute 98 sec latency 37.157 ms 2024-03-24T04:23:17.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 115372 11.92 MB/sec execute 99 sec latency 55.048 ms 2024-03-24T04:23:18.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:18 smithi022 ceph-mon[24756]: pgmap v59: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 666 op/s 2024-03-24T04:23:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:18 smithi116 ceph-mon[27165]: pgmap v59: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 666 op/s 2024-03-24T04:23:18.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 115892 11.90 MB/sec execute 100 sec latency 46.856 ms 2024-03-24T04:23:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 116357 11.85 MB/sec execute 101 sec latency 63.899 ms 2024-03-24T04:23:20.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: pgmap v60: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.2 MiB/s wr, 411 op/s 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: Standby manager daemon smithi022.ijpjvk restarted 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: Standby manager daemon smithi022.ijpjvk started 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/crt"}]: dispatch 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/key"}]: dispatch 2024-03-24T04:23:20.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:20 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:23:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: pgmap v60: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.2 MiB/s wr, 411 op/s 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: Standby manager daemon smithi022.ijpjvk restarted 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: Standby manager daemon smithi022.ijpjvk started 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/crt"}]: dispatch 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi022.ijpjvk/key"}]: dispatch 2024-03-24T04:23:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:20 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:23:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 116908 11.96 MB/sec execute 102 sec latency 43.459 ms 2024-03-24T04:23:21.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:21 smithi022 ceph-mon[24756]: mgrmap e29: smithi116.oxktjt(active, since 115s), standbys: smithi022.ijpjvk 2024-03-24T04:23:21.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:21 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:21.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:21 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:21 smithi116 ceph-mon[27165]: mgrmap e29: smithi116.oxktjt(active, since 115s), standbys: smithi022.ijpjvk 2024-03-24T04:23:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:21 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:21 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:21.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 117510 12.05 MB/sec execute 103 sec latency 29.676 ms 2024-03-24T04:23:22.232 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:22 smithi022 ceph-mon[24756]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.5 MiB/s wr, 555 op/s 2024-03-24T04:23:22.439 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:22 smithi116 ceph-mon[27165]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.5 MiB/s wr, 555 op/s 2024-03-24T04:23:22.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 117862 12.02 MB/sec execute 104 sec latency 57.844 ms 2024-03-24T04:23:23.367 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:23:23.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 118238 11.98 MB/sec execute 105 sec latency 34.292 ms 2024-03-24T04:23:24.288 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:23:24.288 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (82s) 5s ago 10m 25.5M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:23:24.288 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (10m) 5s ago 10m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (9m) 11s ago 9m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (60s) 5s ago 9m 43.4M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (4m) 5s ago 4m 36.2M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (4m) 11s ago 4m 14.4M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (15s) 5s ago 11m 98.8M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (2m) 11s ago 9m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6ddb3fd00fbd 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (11m) 5s ago 11m 50.9M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (9m) 11s ago 9m 42.9M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (3m) 5s ago 3m 254M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (3m) 11s ago 3m 85.5M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (56s) 5s ago 9m 10.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (27s) 11s ago 9m 13.5M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (8m) 5s ago 8m 443M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:23:24.289 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (8m) 5s ago 8m 332M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (7m) 5s ago 7m 297M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (7m) 5s ago 7m 406M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (6m) 11s ago 6m 381M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (6m) 11s ago 6m 422M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (6m) 11s ago 6m 363M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (5m) 11s ago 5m 499M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:23:24.290 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (30s) 5s ago 8m 48.7M - 2.33.4 514e6a882f6e 86fd0f53aab9 2024-03-24T04:23:24.540 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:24 smithi022 ceph-mon[24756]: pgmap v62: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 661 op/s 2024-03-24T04:23:24.541 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:24 smithi022 ceph-mon[24756]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:24 smithi116 ceph-mon[27165]: pgmap v62: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 661 op/s 2024-03-24T04:23:24.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:24 smithi116 ceph-mon[27165]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: 5 118577 11.95 MB/sec execute 106 sec latency 35.459 ms 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:24.956 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:23:24.957 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:23:25.449 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:23:25.449 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:23:25.449 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading mgr daemons", 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:23:25.450 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:23:25.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:25 smithi022 ceph-mon[24756]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:25.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:25 smithi022 ceph-mon[24756]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:25.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:25 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/4090900080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:23:25.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:25 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:25.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:25 smithi116 ceph-mon[27165]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:25.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:25 smithi116 ceph-mon[27165]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:25.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:25 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/4090900080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:23:25.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:25 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:25.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 118919 11.93 MB/sec execute 107 sec latency 75.676 ms 2024-03-24T04:23:26.049 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:23:26.587 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:26 smithi022 ceph-mon[24756]: pgmap v63: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 490 op/s 2024-03-24T04:23:26.587 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:26 smithi022 ceph-mon[24756]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:26.587 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:26 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2263253458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:23:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:26 smithi116 ceph-mon[27165]: pgmap v63: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 490 op/s 2024-03-24T04:23:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:26 smithi116 ceph-mon[27165]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:26 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2263253458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:23:26.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 119552 11.92 MB/sec execute 108 sec latency 40.875 ms 2024-03-24T04:23:27.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 120116 11.95 MB/sec execute 109 sec latency 41.821 ms 2024-03-24T04:23:28.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:28.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: pgmap v64: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 734 op/s 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: Failing over to other MGR 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:28.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:28.378 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T04:23:28.378 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:28 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: pgmap v64: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 734 op/s 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:28.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: Upgrade: Need to upgrade myself (mgr.smithi116.oxktjt) 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: Failing over to other MGR 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 172.21.15.116:0/2595024659' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T04:23:28.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:28 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:23:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 120605 12.01 MB/sec execute 110 sec latency 33.987 ms 2024-03-24T04:23:29.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd='[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]': finished 2024-03-24T04:23:29.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: mgrmap e30: smithi022.ijpjvk(active, starting, since 0.878632s), standbys: smithi116.oxktjt 2024-03-24T04:23:29.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:23:29.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:23:29.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:23:29.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:29 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:23:29.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.24423 ' entity='mgr.smithi116.oxktjt' cmd='[{"prefix": "mgr fail", "who": "smithi116.oxktjt"}]': finished 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: mgrmap e30: smithi022.ijpjvk(active, starting, since 0.878632s), standbys: smithi116.oxktjt 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:23:29.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:29.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:23:29.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:29 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:23:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 121112 12.05 MB/sec execute 111 sec latency 59.032 ms 2024-03-24T04:23:30.431 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:30 smithi022 ceph-mon[24756]: mgrmap e31: smithi022.ijpjvk(active, since 1.89645s), standbys: smithi116.oxktjt 2024-03-24T04:23:30.431 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:30 smithi022 ceph-mon[24756]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:30 smithi116 ceph-mon[27165]: mgrmap e31: smithi022.ijpjvk(active, since 1.89645s), standbys: smithi116.oxktjt 2024-03-24T04:23:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:30 smithi116 ceph-mon[27165]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:30.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 121501 12.03 MB/sec execute 112 sec latency 31.279 ms 2024-03-24T04:23:31.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:31 smithi022 ceph-mon[24756]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:31.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:31 smithi022 ceph-mon[24756]: mgrmap e32: smithi022.ijpjvk(active, since 2s), standbys: smithi116.oxktjt 2024-03-24T04:23:31.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:31 smithi116 ceph-mon[27165]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:31.547 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:31 smithi116 ceph-mon[27165]: mgrmap e32: smithi022.ijpjvk(active, since 2s), standbys: smithi116.oxktjt 2024-03-24T04:23:31.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 121805 11.98 MB/sec execute 113 sec latency 87.591 ms 2024-03-24T04:23:32.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:32 smithi022 ceph-mon[24756]: [24/Mar/2024:04:23:31] ENGINE Bus STARTING 2024-03-24T04:23:32.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:32 smithi022 ceph-mon[24756]: [24/Mar/2024:04:23:31] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T04:23:32.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:32 smithi022 ceph-mon[24756]: [24/Mar/2024:04:23:31] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:23:32.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:32 smithi022 ceph-mon[24756]: [24/Mar/2024:04:23:31] ENGINE Bus STARTED 2024-03-24T04:23:32.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:32 smithi116 ceph-mon[27165]: [24/Mar/2024:04:23:31] ENGINE Bus STARTING 2024-03-24T04:23:32.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:32 smithi116 ceph-mon[27165]: [24/Mar/2024:04:23:31] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T04:23:32.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:32 smithi116 ceph-mon[27165]: [24/Mar/2024:04:23:31] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:23:32.713 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:32 smithi116 ceph-mon[27165]: [24/Mar/2024:04:23:31] ENGINE Bus STARTED 2024-03-24T04:23:32.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 122100 11.95 MB/sec execute 114 sec latency 56.507 ms 2024-03-24T04:23:33.240 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:33 smithi022 ceph-mon[24756]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:33.240 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:33 smithi022 ceph-mon[24756]: mgrmap e33: smithi022.ijpjvk(active, since 4s), standbys: smithi116.oxktjt 2024-03-24T04:23:33.453 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:33 smithi116 ceph-mon[27165]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:33.453 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:33 smithi116 ceph-mon[27165]: mgrmap e33: smithi022.ijpjvk(active, since 4s), standbys: smithi116.oxktjt 2024-03-24T04:23:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 122403 11.93 MB/sec execute 115 sec latency 53.454 ms 2024-03-24T04:23:34.599 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:34 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:34.600 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:34 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:34.689 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:34 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:34.689 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:34 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:34.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 122933 11.88 MB/sec execute 116 sec latency 71.644 ms 2024-03-24T04:23:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 123435 11.86 MB/sec execute 117 sec latency 25.284 ms 2024-03-24T04:23:35.850 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:35 smithi022 ceph-mon[24756]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:35.851 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:35 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:35.851 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:35 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:35.950 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:35 smithi116 ceph-mon[27165]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:23:35.950 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:35 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:35.950 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:35 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:36.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 123963 11.94 MB/sec execute 118 sec latency 29.866 ms 2024-03-24T04:23:37.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 124390 11.98 MB/sec execute 119 sec latency 52.360 ms 2024-03-24T04:23:37.894 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: pgmap v7: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 MiB/s wr, 407 op/s 2024-03-24T04:23:37.894 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:23:37.895 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:23:37.895 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:23:37.895 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:23:37.895 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:23:37.895 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:37 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:23:37.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: pgmap v7: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 MiB/s wr, 407 op/s 2024-03-24T04:23:37.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:23:37.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:23:37.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:23:37.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:23:37.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:23:37.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:37 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/1637936420' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:23:38.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:38 smithi022 ceph-mon[24756]: mgrmap e34: smithi022.ijpjvk(active, since 10s), standbys: smithi116.oxktjt 2024-03-24T04:23:38.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:38 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:38.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:38 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 124763 11.96 MB/sec execute 120 sec latency 31.397 ms 2024-03-24T04:23:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:38 smithi116 ceph-mon[27165]: mgrmap e34: smithi022.ijpjvk(active, since 10s), standbys: smithi116.oxktjt 2024-03-24T04:23:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:38 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:38 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 125146 11.95 MB/sec execute 121 sec latency 44.052 ms 2024-03-24T04:23:39.928 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:39 smithi022 ceph-mon[24756]: pgmap v8: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-03-24T04:23:39.928 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:39 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:39.928 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:39 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:39 smithi116 ceph-mon[27165]: pgmap v8: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-03-24T04:23:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:39 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:39 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:40.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 125544 11.92 MB/sec execute 122 sec latency 28.491 ms 2024-03-24T04:23:41.819 INFO:teuthology.orchestra.run.smithi022.stdout: 5 126025 11.93 MB/sec execute 123 sec latency 41.017 ms 2024-03-24T04:23:41.955 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:41 smithi022 ceph-mon[24756]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 552 op/s 2024-03-24T04:23:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:41 smithi116 ceph-mon[27165]: pgmap v9: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 552 op/s 2024-03-24T04:23:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 126429 11.88 MB/sec execute 124 sec latency 62.871 ms 2024-03-24T04:23:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 126806 11.85 MB/sec execute 125 sec latency 34.969 ms 2024-03-24T04:23:43.993 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:43 smithi022 ceph-mon[24756]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-24T04:23:43.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:43 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:44.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:43 smithi116 ceph-mon[27165]: pgmap v10: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-24T04:23:44.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:43 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:44.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 127195 11.85 MB/sec execute 126 sec latency 54.368 ms 2024-03-24T04:23:45.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 127548 11.85 MB/sec execute 127 sec latency 49.785 ms 2024-03-24T04:23:46.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:45 smithi022 ceph-mon[24756]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-24T04:23:46.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:45 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:46.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:45 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:46.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:45 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:46.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:45 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:23:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:45 smithi116 ceph-mon[27165]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-24T04:23:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:45 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:45 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:45 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:46.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:45 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:23:46.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 127858 11.85 MB/sec execute 128 sec latency 52.673 ms 2024-03-24T04:23:47.057 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:23:47.057 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.058 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:46 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:23:47.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:46 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:47.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 128152 11.85 MB/sec execute 129 sec latency 61.200 ms 2024-03-24T04:23:47.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:47 smithi022 ceph-mon[24756]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:23:47.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:47 smithi022 ceph-mon[24756]: pgmap v12: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 7.4 MiB/s wr, 758 op/s 2024-03-24T04:23:47.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:47 smithi022 ceph-mon[24756]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:23:47.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:47 smithi022 ceph-mon[24756]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T04:23:48.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:47 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:23:48.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:47 smithi116 ceph-mon[27165]: pgmap v12: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 7.4 MiB/s wr, 758 op/s 2024-03-24T04:23:48.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:47 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:23:48.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:47 smithi116 ceph-mon[27165]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T04:23:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 128391 11.80 MB/sec execute 130 sec latency 38.484 ms 2024-03-24T04:23:48.841 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:48 smithi022 ceph-mon[24756]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T04:23:49.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:48 smithi116 ceph-mon[27165]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T04:23:49.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 128754 11.78 MB/sec execute 131 sec latency 47.938 ms 2024-03-24T04:23:49.876 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:49 smithi022 ceph-mon[24756]: pgmap v13: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 523 op/s 2024-03-24T04:23:50.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:49 smithi116 ceph-mon[27165]: pgmap v13: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 523 op/s 2024-03-24T04:23:50.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 129079 11.75 MB/sec execute 132 sec latency 33.895 ms 2024-03-24T04:23:51.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 129452 11.75 MB/sec execute 133 sec latency 38.797 ms 2024-03-24T04:23:51.903 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:51 smithi022 ceph-mon[24756]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 713 op/s 2024-03-24T04:23:52.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:51 smithi116 ceph-mon[27165]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 713 op/s 2024-03-24T04:23:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 129974 11.71 MB/sec execute 134 sec latency 43.466 ms 2024-03-24T04:23:53.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 130424 11.69 MB/sec execute 135 sec latency 41.309 ms 2024-03-24T04:23:53.929 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:53 smithi022 ceph-mon[24756]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 445 op/s 2024-03-24T04:23:54.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:53 smithi116 ceph-mon[27165]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 445 op/s 2024-03-24T04:23:54.820 INFO:teuthology.orchestra.run.smithi022.stdout: 5 130816 11.69 MB/sec execute 136 sec latency 38.918 ms 2024-03-24T04:23:55.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 131150 11.70 MB/sec execute 137 sec latency 77.918 ms 2024-03-24T04:23:55.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:55 smithi022 ceph-mon[24756]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 445 op/s 2024-03-24T04:23:55.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:55 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:55.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:55 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:55.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:55 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:23:55.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:55 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:56.139 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:55 smithi116 ceph-mon[27165]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 445 op/s 2024-03-24T04:23:56.139 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:55 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:56.140 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:55 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:23:56.140 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:55 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:23:56.140 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:55 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:23:56.499 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:23:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 131576 11.72 MB/sec execute 138 sec latency 43.579 ms 2024-03-24T04:23:57.029 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:56 smithi022 ceph-mon[24756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:23:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:56 smithi116 ceph-mon[27165]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (115s) 19s ago 10m 25.5M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (10m) 19s ago 10m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (10m) 23s ago 10m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (93s) 19s ago 10m 45.3M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (4m) 19s ago 4m 36.3M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (4m) 23s ago 4m 14.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (48s) 19s ago 12m 480M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283 running (2m) 23s ago 9m 82.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6ddb3fd00fbd 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (12m) 19s ago 12m 54.4M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (9m) 23s ago 9m 45.0M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (4m) 19s ago 4m 253M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (4m) 23s ago 4m 85.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:23:57.340 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (89s) 19s ago 10m 11.2M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (60s) 23s ago 9m 9.97M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (9m) 19s ago 9m 452M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (8m) 19s ago 8m 389M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (8m) 19s ago 8m 299M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (7m) 19s ago 7m 422M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (7m) 23s ago 7m 406M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (7m) 23s ago 7m 447M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (6m) 23s ago 6m 402M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (6m) 23s ago 6m 515M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:23:57.341 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 starting - - - - 2024-03-24T04:23:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 131813 11.70 MB/sec execute 139 sec latency 80.498 ms 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:23:57.863 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:23:57.864 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:23:57.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: pgmap v17: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 697 op/s 2024-03-24T04:23:57.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:57.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:57.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:57.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:23:57.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:57 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:58.060 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: pgmap v17: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 697 op/s 2024-03-24T04:23:58.060 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:58.060 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:58.061 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:23:58.061 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:23:58.061 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:57 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:23:58.303 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading mgr daemons", 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:23:58.304 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:23:58.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 132112 11.69 MB/sec execute 140 sec latency 81.884 ms 2024-03-24T04:23:58.835 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: Upgrade: Updating mgr.smithi116.oxktjt 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2080276827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:58.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:58 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2625652941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:23:59.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:23:59.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: Upgrade: Updating mgr.smithi116.oxktjt 2024-03-24T04:23:59.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: Deploying daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:23:59.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2080276827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:23:59.078 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:23:59.079 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:58 smithi022 ceph-mon[24756]: from='client.? 172.21.15.22:0/2625652941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:23:59.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 132461 11.64 MB/sec execute 141 sec latency 35.541 ms 2024-03-24T04:23:59.950 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:59 smithi116 ceph-mon[27165]: pgmap v18: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-03-24T04:23:59.950 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:23:59 smithi116 ceph-mon[27165]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:00.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:59 smithi022 ceph-mon[24756]: pgmap v18: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-03-24T04:24:00.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:23:59 smithi022 ceph-mon[24756]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 132811 11.62 MB/sec execute 142 sec latency 39.855 ms 2024-03-24T04:24:01.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 133188 11.62 MB/sec execute 143 sec latency 79.929 ms 2024-03-24T04:24:01.870 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:01 smithi022 ceph-mon[24756]: pgmap v19: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.9 MiB/s wr, 637 op/s 2024-03-24T04:24:01.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:01 smithi116 ceph-mon[27165]: pgmap v19: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.9 MiB/s wr, 637 op/s 2024-03-24T04:24:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 133621 11.56 MB/sec execute 144 sec latency 47.920 ms 2024-03-24T04:24:03.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 133882 11.53 MB/sec execute 145 sec latency 59.244 ms 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: pgmap v20: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:03.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:03 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: pgmap v20: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-03-24T04:24:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:04.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:04.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:04.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:03 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:04.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 134052 11.48 MB/sec execute 146 sec latency 82.339 ms 2024-03-24T04:24:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 134365 11.47 MB/sec execute 147 sec latency 51.225 ms 2024-03-24T04:24:05.913 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:05 smithi116 ceph-mon[27165]: pgmap v21: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-03-24T04:24:05.968 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:05 smithi022 ceph-mon[24756]: pgmap v21: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-03-24T04:24:06.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 134617 11.46 MB/sec execute 148 sec latency 83.384 ms 2024-03-24T04:24:07.801 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:07 smithi022 ceph-mon[24756]: pgmap v22: 97 pgs: 97 active+clean; 324 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 664 op/s 2024-03-24T04:24:07.817 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:07 smithi116 ceph-mon[27165]: pgmap v22: 97 pgs: 97 active+clean; 324 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 664 op/s 2024-03-24T04:24:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 134882 11.42 MB/sec execute 149 sec latency 95.767 ms 2024-03-24T04:24:08.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 135189 11.43 MB/sec execute 150 sec latency 57.240 ms 2024-03-24T04:24:08.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:08 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:08.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:08 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:09.176 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:08 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:09.176 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:08 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:09.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 135492 11.42 MB/sec execute 151 sec latency 47.027 ms 2024-03-24T04:24:09.885 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:09 smithi022 ceph-mon[24756]: pgmap v23: 97 pgs: 97 active+clean; 324 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-24T04:24:09.886 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:09 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:09.886 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:09 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:10.133 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:09 smithi116 ceph-mon[27165]: pgmap v23: 97 pgs: 97 active+clean; 324 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-24T04:24:10.134 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:09 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:10.134 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:09 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:10.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 135873 11.42 MB/sec execute 152 sec latency 34.841 ms 2024-03-24T04:24:11.598 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:11 smithi116 ceph-mon[27165]: pgmap v24: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 576 op/s 2024-03-24T04:24:11.598 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:11 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:11.598 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:11 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:11.660 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:11 smithi022 ceph-mon[24756]: pgmap v24: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 576 op/s 2024-03-24T04:24:11.660 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:11 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:11.660 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:11 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 136162 11.38 MB/sec execute 153 sec latency 75.570 ms 2024-03-24T04:24:12.679 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:24:12.679 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:24:12.680 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:24:12.680 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:24:12.680 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:24:12.680 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:12 smithi022 ceph-mon[24756]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:24:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:24:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:24:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:24:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:24:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:24:12.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:12 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3023344795' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:24:12.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 136440 11.35 MB/sec execute 154 sec latency 39.206 ms 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: pgmap v25: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: mgrmap e35: smithi022.ijpjvk(active, since 45s), standbys: smithi116.oxktjt 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:13.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:13.695 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:13.695 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.695 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:13 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi022.ijpjvk"}]: dispatch 2024-03-24T04:24:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: pgmap v25: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-03-24T04:24:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: mgrmap e35: smithi022.ijpjvk(active, since 45s), standbys: smithi116.oxktjt 2024-03-24T04:24:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:13.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:13 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi022.ijpjvk"}]: dispatch 2024-03-24T04:24:13.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 136807 11.34 MB/sec execute 155 sec latency 71.857 ms 2024-03-24T04:24:14.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:14 smithi022 ceph-mon[24756]: Upgrade: Setting container_image for all mgr 2024-03-24T04:24:14.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:14 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi022.ijpjvk"}]': finished 2024-03-24T04:24:14.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:14 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.oxktjt"}]: dispatch 2024-03-24T04:24:14.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:14 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.oxktjt"}]': finished 2024-03-24T04:24:14.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:14 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T04:24:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:14 smithi116 ceph-mon[27165]: Upgrade: Setting container_image for all mgr 2024-03-24T04:24:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:14 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi022.ijpjvk"}]': finished 2024-03-24T04:24:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:14 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.oxktjt"}]: dispatch 2024-03-24T04:24:14.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:14 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi116.oxktjt"}]': finished 2024-03-24T04:24:14.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:14 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T04:24:14.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 137250 11.30 MB/sec execute 156 sec latency 50.586 ms 2024-03-24T04:24:15.478 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:15 smithi022 ceph-mon[24756]: pgmap v26: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-03-24T04:24:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:15 smithi116 ceph-mon[27165]: pgmap v26: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-03-24T04:24:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: 5 137593 11.28 MB/sec execute 157 sec latency 38.442 ms 2024-03-24T04:24:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 137975 11.30 MB/sec execute 158 sec latency 57.097 ms 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: Upgrade: Updating mon.smithi022 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: Deploying daemon mon.smithi022 on smithi022 2024-03-24T04:24:17.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:17 smithi022 ceph-mon[24756]: pgmap v27: 97 pgs: 97 active+clean; 322 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-03-24T04:24:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: Upgrade: Updating mon.smithi022 2024-03-24T04:24:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:24:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:24:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: Deploying daemon mon.smithi022 on smithi022 2024-03-24T04:24:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:17 smithi116 ceph-mon[27165]: pgmap v27: 97 pgs: 97 active+clean; 322 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-03-24T04:24:17.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 138335 11.29 MB/sec execute 159 sec latency 59.352 ms 2024-03-24T04:24:18.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 138947 11.37 MB/sec execute 160 sec latency 29.238 ms 2024-03-24T04:24:19.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:19 smithi022 ceph-mon[24756]: pgmap v28: 97 pgs: 97 active+clean; 322 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 369 op/s 2024-03-24T04:24:19.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:19 smithi022 systemd[1]: Stopping Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:24:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:19 smithi116 ceph-mon[27165]: pgmap v28: 97 pgs: 97 active+clean; 322 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 369 op/s 2024-03-24T04:24:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 139423 11.37 MB/sec execute 161 sec latency 34.276 ms 2024-03-24T04:24:19.869 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:19 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[24752]: 2024-03-24T04:24:19.813+0000 7fd566081700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi022 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:24:19.869 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:19 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[24752]: 2024-03-24T04:24:19.813+0000 7fd566081700 -1 mon.smithi022@0(leader) e2 *** Got Signal Terminated *** 2024-03-24T04:24:19.869 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:19 smithi022 podman[66179]: 2024-03-24 04:24:19.866984548 +0000 UTC m=+0.195003763 container died 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, GIT_CLEAN=True, release=754, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T04:24:20.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:20 smithi022 podman[66179]: 2024-03-24 04:24:20.669877276 +0000 UTC m=+0.997896476 container cleanup 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, vcs-type=git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-24T04:24:20.795 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:20 smithi022 bash[66179]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022 2024-03-24T04:24:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 139803 11.36 MB/sec execute 162 sec latency 42.543 ms 2024-03-24T04:24:21.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:21 smithi022 podman[66191]: 2024-03-24 04:24:21.279024546 +0000 UTC m=+1.405961408 container remove 5834ec376ce5331afe5fe276b6c93550a08ade55a2a57c9fe9e2f17d446e4db5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vcs-type=git, ceph=True, name=centos-stream, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=) 2024-03-24T04:24:21.802 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:21 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service: Deactivated successfully. 2024-03-24T04:24:21.803 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:21 smithi022 systemd[1]: Stopped Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:24:21.803 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:21 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service: Consumed 16.359s CPU time. 2024-03-24T04:24:21.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 140270 11.37 MB/sec execute 163 sec latency 41.438 ms 2024-03-24T04:24:22.053 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:21 smithi022 systemd[1]: Starting Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:24:22.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 podman[66320]: 2024-03-24 04:24:22.141609378 +0000 UTC m=+0.028538209 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:24:22.304 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 podman[66320]: 2024-03-24 04:24:22.26323189 +0000 UTC m=+0.150160753 container create ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 podman[66320]: 2024-03-24 04:24:22.633964378 +0000 UTC m=+0.520893212 container init ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 podman[66320]: 2024-03-24 04:24:22.642496911 +0000 UTC m=+0.529425740 container start ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: pidfile_write: ignore empty --pid-file 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: load: jerasure load: lrc 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: RocksDB version: 7.9.2 2024-03-24T04:24:22.807 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Git sha 0 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: DB SUMMARY 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: DB Session ID: 0DQF3ZTM1NWQN31TDZ1K 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: CURRENT file: CURRENT 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi022/store.db dir, Total Num: 1, files: 000030.sst 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi022/store.db: 000028.log size: 1515160 ; 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.error_if_exists: 0 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.create_if_missing: 0 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.paranoid_checks: 1 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T04:24:22.808 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T04:24:22.809 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.env: 0x564ea2febae0 2024-03-24T04:24:22.809 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T04:24:22.809 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.info_log: 0x564ea3b5d760 2024-03-24T04:24:22.809 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T04:24:22.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.statistics: (nil) 2024-03-24T04:24:22.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.use_fsync: 0 2024-03-24T04:24:22.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_log_file_size: 0 2024-03-24T04:24:22.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T04:24:22.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_fallocate: 1 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.use_direct_reads: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.db_log_dir: 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.wal_dir: 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T04:24:22.811 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.write_buffer_manager: 0x564ea3b31400 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.unordered_write: 0 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T04:24:22.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.row_cache: None 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.wal_filter: None 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.two_write_queues: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.wal_compression: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.atomic_flush: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.log_readahead_size: 0 2024-03-24T04:24:22.813 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_background_jobs: 2 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_background_compactions: -1 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_subcompactions: 1 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T04:24:22.814 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_open_files: -1 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_background_flushes: -1 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Compression algorithms supported: 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kZSTD supported: 0 2024-03-24T04:24:22.815 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kXpressCompression supported: 0 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kZlibCompression supported: 1 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kSnappyCompression supported: 1 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kLZ4Compression supported: 1 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: kBZip2Compression supported: 0 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000015 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.merge_operator: 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_filter: None 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T04:24:22.816 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564ea3b5d880) 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks: 1 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_top_level_index_and_filter: 1 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_type: 0 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_index_type: 0 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_shortening: 1 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: checksum: 4 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: no_block_cache: 0 2024-03-24T04:24:22.817 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache: 0x564ea3b611f0 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_name: BinnedLRUCache 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_options: 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: capacity : 536870912 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: num_shard_bits : 4 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: strict_capacity_limit : 0 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: high_pri_pool_ratio: 0.000 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_compressed: (nil) 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: persistent_cache: (nil) 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size: 4096 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size_deviation: 10 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_restart_interval: 16 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_block_restart_interval: 1 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: metadata_block_size: 4096 2024-03-24T04:24:22.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: partition_filters: 0 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: use_delta_encoding: 1 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: filter_policy: bloomfilter 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: whole_key_filtering: 1 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: verify_compression: 0 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: read_amp_bytes_per_bit: 0 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: format_version: 5 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_index_compression: 1 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_align: 0 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: max_auto_readahead_size: 262144 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: prepopulate_block_cache: 0 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: initial_auto_readahead_size: 8192 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression: NoCompression 2024-03-24T04:24:22.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.num_levels: 7 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T04:24:22.820 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T04:24:22.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T04:24:22.822 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.table_properties_collectors: 2024-03-24T04:24:22.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.inplace_update_support: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.bloom_locality: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.max_successive_merges: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.ttl: 2592000 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enable_blob_files: false 2024-03-24T04:24:22.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.min_blob_size: 0 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11893, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T04:24:22.825 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6d411d2c-f80f-48b0-917d-e903304978f2 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254262698737, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254262707934, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 1347850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11887, "largest_seqno": 12705, "table_properties": {"data_size": 1343503, "index_size": 2437, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 901, "raw_key_size": 7946, "raw_average_key_size": 23, "raw_value_size": 1335822, "raw_average_value_size": 3894, "num_data_blocks": 112, "num_entries": 343, "num_filter_entries": 343, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711254262, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6d411d2c-f80f-48b0-917d-e903304978f2", "db_session_id": "0DQF3ZTM1NWQN31TDZ1K", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254262708049, "job": 1, "event": "recovery_finished"} 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi022/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564ea3c50000 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: DB pointer 0x564ea3c3c000 2024-03-24T04:24:22.826 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: starting mon.smithi022 rank 0 at public addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] at bind addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi022 fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** DB Stats ** 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T04:24:22.827 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: L0 1/0 1.29 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 314.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: L6 1/0 7.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Sum 2/0 8.86 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 314.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 314.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 314.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:24:22.828 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative compaction: 0.00 GB write, 65.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval compaction: 0.00 GB write, 65.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Block cache BinnedLRUCache@0x564ea3b611f0#2 capacity: 512.00 MB usage: 46.45 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 6.9e-05 secs_since: 0 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Block cache entry stats(count,size,portion): DataBlock(3,17.73 KB,0.00338256%) FilterBlock(2,10.06 KB,0.00191927%) IndexBlock(2,18.66 KB,0.0035584%) Misc(1,0.00 KB,0%) 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???) e2 preinit fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).mds e7 new map 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).mds e7 print_map 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: e7 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T04:24:22.829 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: legacy client fscid: 1 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Filesystem 'foofs' (1) 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: fs_name foofs 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: epoch 5 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: created 2024-03-24T04:19:02.269434+0000 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: modified 2024-03-24T04:19:08.305481+0000 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: tableserver 0 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: root 0 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: session_timeout 60 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: session_autoclose 300 2024-03-24T04:24:22.830 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: max_file_size 1099511627776 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: max_xattr_size 65536 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: required_client_features {} 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: last_failure 0 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: last_failure_osd_epoch 0 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: max_mds 1 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: in 0 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: up {0=14504} 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: failed 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: damaged 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: stopped 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_pools [3] 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: metadata_pool 2 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: inline_data disabled 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: balancer 2024-03-24T04:24:22.831 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: bal_rank_mask -1 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: standby_count_wanted 1 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: [mds.foofs.smithi022.ekdqmp{0:14504} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Standby daemons: 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: [mds.foofs.smithi116.kvgpsf{-1:24283} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).osd e71 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:22.832 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 ceph-mon[66341]: mon.smithi022@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-03-24T04:24:22.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 140881 11.35 MB/sec execute 164 sec latency 41.293 ms 2024-03-24T04:24:23.063 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 bash[66320]: ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:22 smithi022 systemd[1]: Started Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: mon.smithi022 calling monitor election 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: mgrmap e35: smithi022.ijpjvk(active, since 55s), standbys: smithi116.oxktjt 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: overall HEALTH_OK 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: from='mgr.14792 ' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: mgrmap e36: smithi022.ijpjvk(active, since 55s), standbys: smithi116.oxktjt 2024-03-24T04:24:23.064 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:23 smithi022 ceph-mon[66341]: from='mgr.14792 ' entity='' 2024-03-24T04:24:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: from='mgr.14792 172.21.15.22:0/184541205' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: mon.smithi022 calling monitor election 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: mgrmap e35: smithi022.ijpjvk(active, since 55s), standbys: smithi116.oxktjt 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: overall HEALTH_OK 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: from='mgr.14792 ' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: mgrmap e36: smithi022.ijpjvk(active, since 55s), standbys: smithi116.oxktjt 2024-03-24T04:24:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:23 smithi116 ceph-mon[27165]: from='mgr.14792 ' entity='' 2024-03-24T04:24:23.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 141380 11.36 MB/sec execute 165 sec latency 44.307 ms 2024-03-24T04:24:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 141930 11.39 MB/sec execute 166 sec latency 38.611 ms 2024-03-24T04:24:25.083 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:25 smithi022 ceph-mon[66341]: mgrmap e37: smithi022.ijpjvk(active, since 56s), standbys: smithi116.oxktjt 2024-03-24T04:24:25.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:25 smithi116 ceph-mon[27165]: mgrmap e37: smithi022.ijpjvk(active, since 56s), standbys: smithi116.oxktjt 2024-03-24T04:24:25.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 142393 11.44 MB/sec execute 167 sec latency 33.220 ms 2024-03-24T04:24:26.822 INFO:teuthology.orchestra.run.smithi022.stdout: 5 142690 11.44 MB/sec execute 168 sec latency 39.812 ms 2024-03-24T04:24:27.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 143029 11.41 MB/sec execute 169 sec latency 42.233 ms 2024-03-24T04:24:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 143346 11.39 MB/sec execute 170 sec latency 76.419 ms 2024-03-24T04:24:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 143715 11.38 MB/sec execute 171 sec latency 44.893 ms 2024-03-24T04:24:30.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 144156 11.34 MB/sec execute 172 sec latency 40.475 ms 2024-03-24T04:24:31.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:24:31.248 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:24:31.248 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:24:31.248 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:24:31.248 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:24:31.248 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:31 smithi022 ceph-mon[66341]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:24:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt restarted 2024-03-24T04:24:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: Standby manager daemon smithi116.oxktjt started 2024-03-24T04:24:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/crt"}]: dispatch 2024-03-24T04:24:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T04:24:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi116.oxktjt/key"}]: dispatch 2024-03-24T04:24:31.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:31 smithi116 ceph-mon[27165]: from='mgr.? 172.21.15.116:0/3459831478' entity='mgr.smithi116.oxktjt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T04:24:31.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 144587 11.33 MB/sec execute 173 sec latency 48.493 ms 2024-03-24T04:24:32.265 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:32 smithi022 ceph-mon[66341]: mgrmap e38: smithi022.ijpjvk(active, since 63s), standbys: smithi116.oxktjt 2024-03-24T04:24:32.704 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:32 smithi116 ceph-mon[27165]: mgrmap e38: smithi022.ijpjvk(active, since 63s), standbys: smithi116.oxktjt 2024-03-24T04:24:32.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 145027 11.35 MB/sec execute 174 sec latency 42.853 ms 2024-03-24T04:24:33.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:24:33.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:24:33.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T04:24:33.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: mgrmap e39: smithi022.ijpjvk(active, starting, since 0.00770928s), standbys: smithi116.oxktjt 2024-03-24T04:24:33.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:24:33.291 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: failed to encode map e72 with expected crc 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:24:33.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116[27147]: 2024-03-24T04:24:33.020+0000 7f0161d80700 -1 mon.smithi116@1(peon).osd e72 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: Active manager daemon smithi022.ijpjvk restarted 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: Activating manager daemon smithi022.ijpjvk 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: mgrmap e39: smithi022.ijpjvk(active, starting, since 0.00770928s), standbys: smithi116.oxktjt 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:33.299 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi022.ijpjvk", "id": "smithi022.ijpjvk"}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr metadata", "who": "smithi116.oxktjt", "id": "smithi116.oxktjt"}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:24:33.300 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: Manager daemon smithi022.ijpjvk is now available 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: failed to encode map e72 with expected crc 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/mirror_snapshot_schedule"}]: dispatch 2024-03-24T04:24:33.301 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:33 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.ijpjvk/trash_purge_schedule"}]: dispatch 2024-03-24T04:24:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 145518 11.36 MB/sec execute 175 sec latency 48.301 ms 2024-03-24T04:24:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:24:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: [24/Mar/2024:04:24:33] ENGINE Bus STARTING 2024-03-24T04:24:34.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: mgrmap e40: smithi022.ijpjvk(active, since 1.01375s), standbys: smithi116.oxktjt 2024-03-24T04:24:34.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:34 smithi022 ceph-mon[66341]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: failed to encode map e72 with expected crc 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: [24/Mar/2024:04:24:33] ENGINE Bus STARTING 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: mgrmap e40: smithi022.ijpjvk(active, since 1.01375s), standbys: smithi116.oxktjt 2024-03-24T04:24:34.556 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:34 smithi116 ceph-mon[27165]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:34.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 145964 11.40 MB/sec execute 176 sec latency 54.716 ms 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (2m) 24s ago 11m 25.6M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (11m) 24s ago 11m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (10m) 27s ago 10m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (2m) 24s ago 11m 45.2M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (5m) 24s ago 5m 37.0M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:24:35.167 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (5m) 27s ago 5m 14.8M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (86s) 24s ago 12m 499M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (33s) 27s ago 10m 89.7M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (12m) 24s ago 12m 54.0M 2048M 17.2.0 e1d6a67b021e 5834ec376ce5 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (10m) 27s ago 10m 44.6M 2048M 17.2.0 e1d6a67b021e e42cb14d3d2a 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (5m) 24s ago 5m 261M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (5m) 27s ago 5m 87.4M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (2m) 24s ago 10m 11.0M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (98s) 27s ago 10m 12.8M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (9m) 24s ago 9m 489M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (9m) 24s ago 9m 407M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (8m) 24s ago 8m 320M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (8m) 24s ago 8m 449M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (8m) 27s ago 8m 473M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (7m) 27s ago 7m 504M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (7m) 27s ago 7m 424M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:24:35.168 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (6m) 27s ago 6m 574M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:24:35.169 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (40s) 24s ago 10m 47.4M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:24:35.768 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:24:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:24:35.770 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:24:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 146284 11.41 MB/sec execute 177 sec latency 53.603 ms 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: pgmap v3: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: [24/Mar/2024:04:24:34] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: [24/Mar/2024:04:24:34] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: [24/Mar/2024:04:24:34] ENGINE Bus STARTED 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: pgmap v4: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:36.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:36 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2532024905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:36.209 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:24:36.209 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:24:36.209 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "", 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:24:36.210 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: pgmap v3: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: [24/Mar/2024:04:24:34] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: [24/Mar/2024:04:24:34] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: [24/Mar/2024:04:24:34] ENGINE Bus STARTED 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: pgmap v4: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:36 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/2532024905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:36.820 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:24:36.823 INFO:teuthology.orchestra.run.smithi022.stdout: 5 146538 11.37 MB/sec execute 178 sec latency 63.110 ms 2024-03-24T04:24:37.154 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:37 smithi022 ceph-mon[66341]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:37.154 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:37 smithi022 ceph-mon[66341]: mgrmap e41: smithi022.ijpjvk(active, since 3s), standbys: smithi116.oxktjt 2024-03-24T04:24:37.154 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:37 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1040668369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:24:37.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:37 smithi116 ceph-mon[27165]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:37.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:37 smithi116 ceph-mon[27165]: mgrmap e41: smithi022.ijpjvk(active, since 3s), standbys: smithi116.oxktjt 2024-03-24T04:24:37.369 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:37 smithi116 ceph-mon[27165]: from='client.? 172.21.15.22:0/1040668369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:24:37.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 146787 11.33 MB/sec execute 179 sec latency 68.890 ms 2024-03-24T04:24:38.189 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:38 smithi022 ceph-mon[66341]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:38.189 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:38 smithi022 ceph-mon[66341]: pgmap v5: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:38.189 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:38 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:38.189 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:38 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:38 smithi116 ceph-mon[27165]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:24:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:38 smithi116 ceph-mon[27165]: pgmap v5: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:38 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:38 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 147004 11.29 MB/sec execute 180 sec latency 72.706 ms 2024-03-24T04:24:39.214 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:39 smithi022 ceph-mon[66341]: mgrmap e42: smithi022.ijpjvk(active, since 5s), standbys: smithi116.oxktjt 2024-03-24T04:24:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:39 smithi116 ceph-mon[27165]: mgrmap e42: smithi022.ijpjvk(active, since 5s), standbys: smithi116.oxktjt 2024-03-24T04:24:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 147432 11.30 MB/sec execute 181 sec latency 44.880 ms 2024-03-24T04:24:40.074 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:40 smithi022 ceph-mon[66341]: pgmap v6: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:40.074 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:40 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:40.074 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:40 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:40 smithi116 ceph-mon[27165]: pgmap v6: 97 pgs: 97 active+clean; 322 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-03-24T04:24:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:40 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:40 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:40.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 148054 11.28 MB/sec execute 182 sec latency 31.085 ms 2024-03-24T04:24:41.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 148501 11.29 MB/sec execute 183 sec latency 31.734 ms 2024-03-24T04:24:42.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:42 smithi022 ceph-mon[66341]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-03-24T04:24:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:42 smithi116 ceph-mon[27165]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-03-24T04:24:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 149019 11.32 MB/sec execute 184 sec latency 30.756 ms 2024-03-24T04:24:43.515 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.515 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.515 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.515 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.515 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:43 smithi022 ceph-mon[66341]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.8 MiB/s wr, 260 op/s 2024-03-24T04:24:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:43 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:43 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:43 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:43 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:43.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:43 smithi116 ceph-mon[27165]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.8 MiB/s wr, 260 op/s 2024-03-24T04:24:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 149448 11.36 MB/sec execute 185 sec latency 32.374 ms 2024-03-24T04:24:44.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 149776 11.36 MB/sec execute 186 sec latency 35.975 ms 2024-03-24T04:24:45.037 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:45.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:45.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:45.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:24:45.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:24:45.038 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:44 smithi022 ceph-mon[66341]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:24:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:45.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:45.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:45.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:24:45.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T04:24:45.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:44 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T04:24:45.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 150098 11.33 MB/sec execute 187 sec latency 42.198 ms 2024-03-24T04:24:46.065 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: pgmap v9: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 523 op/s 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.066 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.067 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.067 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:46.067 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:46.067 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:45 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T04:24:46.222 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:24:46.222 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T04:24:46.222 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:24:46.222 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T04:24:46.222 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: Updating smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: pgmap v9: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 523 op/s 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:24:46.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:45 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T04:24:46.826 INFO:teuthology.orchestra.run.smithi022.stdout: 5 150387 11.30 MB/sec execute 188 sec latency 40.914 ms 2024-03-24T04:24:47.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 150625 11.27 MB/sec execute 189 sec latency 46.900 ms 2024-03-24T04:24:48.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:48 smithi022 ceph-mon[66341]: pgmap v10: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 477 op/s 2024-03-24T04:24:48.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:48.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:24:48.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:24:48.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:48.146 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:48 smithi116 ceph-mon[27165]: pgmap v10: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 477 op/s 2024-03-24T04:24:48.147 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:48 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:48.147 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:48 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:24:48.147 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:48 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:24:48.147 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:48 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:24:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 150940 11.26 MB/sec execute 190 sec latency 68.204 ms 2024-03-24T04:24:49.133 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:49 smithi022 ceph-mon[66341]: Upgrade: Updating mon.smithi116 2024-03-24T04:24:49.133 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:49 smithi022 ceph-mon[66341]: Deploying daemon mon.smithi116 on smithi116 2024-03-24T04:24:49.133 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:49.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 ceph-mon[27165]: Upgrade: Updating mon.smithi116 2024-03-24T04:24:49.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 ceph-mon[27165]: Deploying daemon mon.smithi116 on smithi116 2024-03-24T04:24:49.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 ceph-mon[27165]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:24:49.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 systemd[1]: Stopping Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:24:49.827 INFO:teuthology.orchestra.run.smithi022.stdout: 5 151318 11.22 MB/sec execute 191 sec latency 57.334 ms 2024-03-24T04:24:50.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116[27147]: 2024-03-24T04:24:49.895+0000 7f016758b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi116 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:24:50.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116[27147]: 2024-03-24T04:24:49.895+0000 7f016758b700 -1 mon.smithi116@1(peon) e2 *** Got Signal Terminated *** 2024-03-24T04:24:50.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:49 smithi116 podman[53094]: 2024-03-24 04:24:49.937689738 +0000 UTC m=+0.167557344 container died e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, version=8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, name=centos-stream) 2024-03-24T04:24:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 podman[53094]: 2024-03-24 04:24:50.555404048 +0000 UTC m=+0.785271650 container cleanup e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, name=centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-03-24T04:24:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 bash[53094]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116 2024-03-24T04:24:50.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 151671 11.20 MB/sec execute 192 sec latency 48.454 ms 2024-03-24T04:24:51.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 podman[53107]: 2024-03-24 04:24:50.833243211 +0000 UTC m=+0.896932371 container remove e42cb14d3d2a3d7c7e30022737d7cbbd3a3c9b4c04aee8f530909c061ace1681 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, version=8, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, RELEASE=HEAD, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD) 2024-03-24T04:24:51.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi116.service: Deactivated successfully. 2024-03-24T04:24:51.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 systemd[1]: Stopped Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:24:51.174 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:50 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi116.service: Consumed 6.870s CPU time. 2024-03-24T04:24:51.443 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 systemd[1]: Starting Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:24:51.443 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 podman[53232]: 2024-03-24 04:24:51.315992552 +0000 UTC m=+0.024504831 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:24:51.443 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 podman[53232]: 2024-03-24 04:24:51.443124237 +0000 UTC m=+0.151636500 container create 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T04:24:51.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 152193 11.25 MB/sec execute 193 sec latency 64.762 ms 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 podman[53232]: 2024-03-24 04:24:51.729866141 +0000 UTC m=+0.438378404 container init 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 podman[53232]: 2024-03-24 04:24:51.733424187 +0000 UTC m=+0.441936458 container start 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: pidfile_write: ignore empty --pid-file 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: load: jerasure load: lrc 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: RocksDB version: 7.9.2 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Git sha 0 2024-03-24T04:24:52.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: DB SUMMARY 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: DB Session ID: A4578C8CIB0Y6OWX3XZ6 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: CURRENT file: CURRENT 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: MANIFEST file: MANIFEST-000009 size: 905 Bytes 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi116/store.db dir, Total Num: 1, files: 000024.sst 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi116/store.db: 000022.log size: 5494625 ; 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.error_if_exists: 0 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.create_if_missing: 0 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.paranoid_checks: 1 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T04:24:52.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.env: 0x561d72146ae0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.info_log: 0x561d74129760 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.statistics: (nil) 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.use_fsync: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_log_file_size: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_fallocate: 1 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T04:24:52.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.use_direct_reads: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.db_log_dir: 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.wal_dir: 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T04:24:52.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.write_buffer_manager: 0x561d740fd400 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.unordered_write: 0 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.row_cache: None 2024-03-24T04:24:52.228 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.wal_filter: None 2024-03-24T04:24:52.229 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.two_write_queues: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.wal_compression: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.atomic_flush: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.log_readahead_size: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T04:24:52.230 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_background_jobs: 2 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_background_compactions: -1 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_subcompactions: 1 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_open_files: -1 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T04:24:52.231 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_background_flushes: -1 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Compression algorithms supported: 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kZSTD supported: 0 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kXpressCompression supported: 0 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kZlibCompression supported: 1 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kSnappyCompression supported: 1 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kLZ4Compression supported: 1 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: kBZip2Compression supported: 0 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000009 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T04:24:52.232 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.merge_operator: 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_filter: None 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561d74129880) 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_type: 0 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_index_type: 0 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_shortening: 1 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T04:24:52.233 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: checksum: 4 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: no_block_cache: 0 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache: 0x561d7412d090 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_options: 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: capacity : 536870912 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_shard_bits : 4 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: strict_capacity_limit : 0 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_cache_compressed: (nil) 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: persistent_cache: (nil) 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size: 4096 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_size_deviation: 10 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_restart_interval: 16 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: index_block_restart_interval: 1 2024-03-24T04:24:52.234 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_block_size: 4096 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: partition_filters: 0 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: use_delta_encoding: 1 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: filter_policy: bloomfilter 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: whole_key_filtering: 1 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: verify_compression: 0 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: format_version: 5 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_index_compression: 1 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: block_align: 0 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: prepopulate_block_cache: 0 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T04:24:52.235 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression: NoCompression 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.num_levels: 7 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T04:24:52.236 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T04:24:52.237 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T04:24:52.238 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.table_properties_collectors: 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.inplace_update_support: 0 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T04:24:52.239 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.bloom_locality: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.max_successive_merges: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.ttl: 2592000 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enable_blob_files: false 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.min_blob_size: 0 2024-03-24T04:24:52.240 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi116/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12272, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 98eb171d-6008-4e52-be89-4ff387b0525a 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254291766121, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-03-24T04:24:52.241 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254291781821, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 3385382, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12269, "largest_seqno": 13727, "table_properties": {"data_size": 3378801, "index_size": 4090, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 13305, "raw_average_key_size": 23, "raw_value_size": 3366177, "raw_average_value_size": 5854, "num_data_blocks": 187, "num_entries": 575, "num_filter_entries": 575, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711254291, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "98eb171d-6008-4e52-be89-4ff387b0525a", "db_session_id": "A4578C8CIB0Y6OWX3XZ6", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711254291781912, "job": 1, "event": "recovery_finished"} 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi116/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561d7421c000 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: DB pointer 0x561d74208000 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** DB Stats ** 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:52.242 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L0 1/0 3.23 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 490.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: L6 1/0 7.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Sum 2/0 10.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 490.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 490.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T04:24:52.243 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 490.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Cumulative compaction: 0.00 GB write, 166.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Interval compaction: 0.00 GB write, 166.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T04:24:52.244 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache BinnedLRUCache@0x561d7412d090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: starting mon.smithi116 rank 1 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi116 fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???) e2 preinit fsid 8f0be64c-e994-11ee-95cd-87774f69a715 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).mds e7 new map 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).mds e7 print_map 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: e7 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: legacy client fscid: 1 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Filesystem 'foofs' (1) 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: fs_name foofs 2024-03-24T04:24:52.245 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: epoch 5 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: created 2024-03-24T04:19:02.269434+0000 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: modified 2024-03-24T04:19:08.305481+0000 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: tableserver 0 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: root 0 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: session_timeout 60 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: session_autoclose 300 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_file_size 1099511627776 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_xattr_size 65536 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: required_client_features {} 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: last_failure 0 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: last_failure_osd_epoch 0 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T04:24:52.246 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: max_mds 1 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: in 0 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: up {0=14504} 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: failed 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: damaged 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: stopped 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: data_pools [3] 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: metadata_pool 2 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: inline_data disabled 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: balancer 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: bal_rank_mask -1 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: standby_count_wanted 1 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.foofs.smithi022.ekdqmp{0:14504} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.22:6834/2180081849,v1:172.21.15.22:6835/2180081849] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.247 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: Standby daemons: 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout: [mds.foofs.smithi116.kvgpsf{-1:24283} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 ceph-mon[53251]: mon.smithi116@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 bash[53232]: 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 2024-03-24T04:24:52.248 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:51 smithi116 systemd[1]: Started Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:24:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 152602 11.23 MB/sec execute 194 sec latency 54.288 ms 2024-03-24T04:24:53.029 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:53 smithi022 ceph-mon[66341]: pgmap v11: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 477 op/s 2024-03-24T04:24:53.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:53 smithi116 ceph-mon[53251]: pgmap v11: 97 pgs: 97 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 477 op/s 2024-03-24T04:24:53.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 153025 11.27 MB/sec execute 195 sec latency 36.999 ms 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 484 op/s 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: mon.smithi022 calling monitor election 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: mon.smithi116 calling monitor election 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: monmap e3: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T04:24:54.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: mgrmap e42: smithi022.ijpjvk(active, since 20s), standbys: smithi116.oxktjt 2024-03-24T04:24:54.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: overall HEALTH_OK 2024-03-24T04:24:54.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:54.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:54.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:54.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.7 MiB/s wr, 484 op/s 2024-03-24T04:24:54.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T04:24:54.432 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mon metadata", "id": "smithi116"}]: dispatch 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: mon.smithi022 calling monitor election 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: mon.smithi116 calling monitor election 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: mon.smithi022 is new leader, mons smithi022,smithi116 in quorum (ranks 0,1) 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: monmap e3: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi116=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: mgrmap e42: smithi022.ijpjvk(active, since 20s), standbys: smithi116.oxktjt 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: overall HEALTH_OK 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:54.433 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:24:54.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 153515 11.29 MB/sec execute 196 sec latency 32.399 ms 2024-03-24T04:24:55.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 153939 11.27 MB/sec execute 197 sec latency 25.239 ms 2024-03-24T04:24:56.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:56 smithi022 ceph-mon[66341]: pgmap v14: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 727 op/s 2024-03-24T04:24:56.343 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:56 smithi116 ceph-mon[53251]: pgmap v14: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 727 op/s 2024-03-24T04:24:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 154296 11.26 MB/sec execute 198 sec latency 31.166 ms 2024-03-24T04:24:57.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:57 smithi022 ceph-mon[66341]: pgmap v15: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-03-24T04:24:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 154675 11.24 MB/sec execute 199 sec latency 59.781 ms 2024-03-24T04:24:58.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:57 smithi116 ceph-mon[53251]: pgmap v15: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-03-24T04:24:58.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 155123 11.22 MB/sec execute 200 sec latency 39.857 ms 2024-03-24T04:24:59.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:59.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:59.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:59.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:24:59.828 INFO:teuthology.orchestra.run.smithi022.stdout: 5 155583 11.23 MB/sec execute 201 sec latency 36.212 ms 2024-03-24T04:25:00.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:59 smithi022 ceph-mon[66341]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 465 op/s 2024-03-24T04:25:00.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:59 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:00.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:24:59 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:00.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:59 smithi116 ceph-mon[53251]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 465 op/s 2024-03-24T04:25:00.374 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:59 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:00.375 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:24:59 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 156154 11.26 MB/sec execute 202 sec latency 40.527 ms 2024-03-24T04:25:01.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 156681 11.31 MB/sec execute 203 sec latency 28.133 ms 2024-03-24T04:25:02.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:02 smithi022 ceph-mon[66341]: pgmap v17: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 705 op/s 2024-03-24T04:25:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:02 smithi116 ceph-mon[53251]: pgmap v17: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 705 op/s 2024-03-24T04:25:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 157087 11.32 MB/sec execute 204 sec latency 60.366 ms 2024-03-24T04:25:03.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 157472 11.30 MB/sec execute 205 sec latency 42.866 ms 2024-03-24T04:25:04.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: pgmap v18: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 504 op/s 2024-03-24T04:25:04.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.309 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:25:04.310 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:25:04.310 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: pgmap v18: 97 pgs: 97 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 504 op/s 2024-03-24T04:25:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:25:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:04.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 157991 11.31 MB/sec execute 206 sec latency 33.909 ms 2024-03-24T04:25:05.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:05 smithi022 ceph-mon[66341]: Reconfiguring mon.smithi022 (monmap changed)... 2024-03-24T04:25:05.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:05 smithi022 ceph-mon[66341]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T04:25:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:05 smithi116 ceph-mon[53251]: Reconfiguring mon.smithi022 (monmap changed)... 2024-03-24T04:25:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:05 smithi116 ceph-mon[53251]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T04:25:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 158604 11.29 MB/sec execute 207 sec latency 52.617 ms 2024-03-24T04:25:06.325 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:06 smithi022 ceph-mon[66341]: pgmap v19: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 788 op/s 2024-03-24T04:25:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:06 smithi116 ceph-mon[53251]: pgmap v19: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 788 op/s 2024-03-24T04:25:06.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 159214 11.33 MB/sec execute 208 sec latency 43.927 ms 2024-03-24T04:25:07.437 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: Reconfiguring mgr.smithi022.ijpjvk (monmap changed)... 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: Reconfiguring daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:25:07.598 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:07 smithi022 ceph-mon[66341]: pgmap v20: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-03-24T04:25:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: Reconfiguring mgr.smithi022.ijpjvk (monmap changed)... 2024-03-24T04:25:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.ijpjvk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:25:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:25:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: Reconfiguring daemon mgr.smithi022.ijpjvk on smithi022 2024-03-24T04:25:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:07 smithi116 ceph-mon[53251]: pgmap v20: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-03-24T04:25:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 159752 11.35 MB/sec execute 209 sec latency 26.589 ms 2024-03-24T04:25:08.594 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (3m) 26s ago 11m 25.6M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (11m) 26s ago 11m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (11m) 10s ago 11m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (2m) 26s ago 11m 52.8M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (6m) 26s ago 6m 36.9M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (5m) 10s ago 5m 15.5M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (2m) 26s ago 13m 477M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:25:08.595 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (66s) 10s ago 11m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (46s) 26s ago 13m 31.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (17s) 10s ago 11m 21.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (5m) 26s ago 5m 265M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (5m) 10s ago 5m 87.6M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (2m) 26s ago 11m 11.7M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (2m) 10s ago 10m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (10m) 26s ago 10m 511M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (9m) 26s ago 9m 452M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (9m) 26s ago 9m 348M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:25:08.596 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (9m) 26s ago 9m 480M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:25:08.597 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (8m) 10s ago 8m 507M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:25:08.597 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (8m) 10s ago 8m 571M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:25:08.597 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (7m) 10s ago 7m 503M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:25:08.597 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (7m) 10s ago 7m 374M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:25:08.597 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (74s) 26s ago 10m 55.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:25:08.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 160278 11.41 MB/sec execute 210 sec latency 50.756 ms 2024-03-24T04:25:08.861 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:08 smithi022 ceph-mon[66341]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:08.861 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:08 smithi022 ceph-mon[66341]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:09.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:08 smithi116 ceph-mon[53251]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:09.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:08 smithi116 ceph-mon[53251]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:09.245 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:25:09.245 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:25:09.245 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:25:09.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:09.247 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:25:09.247 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T04:25:09.247 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T04:25:09.247 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:25:09.247 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:25:09.743 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: pgmap v21: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3096770640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:09.744 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:09 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:25:09.814 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:25:09.815 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:25:09.815 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "4/23 daemons upgraded", 2024-03-24T04:25:09.815 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading mon daemons", 2024-03-24T04:25:09.815 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:25:09.815 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:25:09.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 160681 11.40 MB/sec execute 211 sec latency 51.460 ms 2024-03-24T04:25:09.865 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: pgmap v21: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3096770640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:09.866 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:09 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:10.491 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:25:10.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 160968 11.37 MB/sec execute 212 sec latency 36.750 ms 2024-03-24T04:25:10.870 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:10 smithi022 ceph-mon[66341]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T04:25:10.870 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:10 smithi022 ceph-mon[66341]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T04:25:10.871 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:10 smithi022 ceph-mon[66341]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:10.871 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:10 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/194593619' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:25:11.193 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:10 smithi116 ceph-mon[53251]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T04:25:11.193 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:10 smithi116 ceph-mon[53251]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T04:25:11.193 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:10 smithi116 ceph-mon[53251]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:11.193 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:10 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/194593619' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:25:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 161239 11.34 MB/sec execute 213 sec latency 47.083 ms 2024-03-24T04:25:11.884 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:11 smithi022 ceph-mon[66341]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 858 op/s 2024-03-24T04:25:11.884 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:11.884 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:12.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:11 smithi116 ceph-mon[53251]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 858 op/s 2024-03-24T04:25:12.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:12.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:12.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 161553 11.34 MB/sec execute 214 sec latency 88.827 ms 2024-03-24T04:25:12.946 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:12 smithi022 ceph-mon[66341]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T04:25:12.946 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:12 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:25:12.946 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:12 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:12.946 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:12 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.0 on smithi022 2024-03-24T04:25:13.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:12 smithi116 ceph-mon[53251]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T04:25:13.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:12 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:25:13.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:12 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:13.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:12 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.0 on smithi022 2024-03-24T04:25:13.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 161923 11.30 MB/sec execute 215 sec latency 50.438 ms 2024-03-24T04:25:13.964 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:13 smithi022 ceph-mon[66341]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.6 MiB/s wr, 619 op/s 2024-03-24T04:25:14.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:13 smithi116 ceph-mon[53251]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.6 MiB/s wr, 619 op/s 2024-03-24T04:25:14.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 162274 11.29 MB/sec execute 216 sec latency 38.126 ms 2024-03-24T04:25:15.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.1 on smithi022 2024-03-24T04:25:15.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:15 smithi022 ceph-mon[66341]: pgmap v24: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 829 op/s 2024-03-24T04:25:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T04:25:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:25:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.1 on smithi022 2024-03-24T04:25:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:15 smithi116 ceph-mon[53251]: pgmap v24: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 829 op/s 2024-03-24T04:25:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 162561 11.27 MB/sec execute 217 sec latency 68.771 ms 2024-03-24T04:25:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 162959 11.28 MB/sec execute 218 sec latency 52.196 ms 2024-03-24T04:25:17.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 163486 11.31 MB/sec execute 219 sec latency 29.585 ms 2024-03-24T04:25:18.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:18 smithi022 ceph-mon[66341]: pgmap v25: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 544 op/s 2024-03-24T04:25:18.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:25:18.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:18 smithi116 ceph-mon[53251]: pgmap v25: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 544 op/s 2024-03-24T04:25:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:25:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:18.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 163918 11.33 MB/sec execute 220 sec latency 50.159 ms 2024-03-24T04:25:19.305 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:19 smithi022 ceph-mon[66341]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T04:25:19.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:19 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.2 on smithi022 2024-03-24T04:25:19.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:19 smithi116 ceph-mon[53251]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T04:25:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:19 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.2 on smithi022 2024-03-24T04:25:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 164389 11.33 MB/sec execute 221 sec latency 43.391 ms 2024-03-24T04:25:20.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:20 smithi022 ceph-mon[66341]: pgmap v26: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-24T04:25:20.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:20.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:20.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:25:20.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:20 smithi116 ceph-mon[53251]: pgmap v26: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-24T04:25:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:25:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 164762 11.31 MB/sec execute 222 sec latency 48.122 ms 2024-03-24T04:25:21.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:21 smithi022 ceph-mon[66341]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T04:25:21.327 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:21 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.3 on smithi022 2024-03-24T04:25:21.327 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:21 smithi022 ceph-mon[66341]: pgmap v27: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 782 op/s 2024-03-24T04:25:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:21 smithi116 ceph-mon[53251]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T04:25:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:21 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.3 on smithi022 2024-03-24T04:25:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:21 smithi116 ceph-mon[53251]: pgmap v27: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 782 op/s 2024-03-24T04:25:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 165281 11.31 MB/sec execute 223 sec latency 48.385 ms 2024-03-24T04:25:22.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 165795 11.30 MB/sec execute 224 sec latency 102.781 ms 2024-03-24T04:25:23.596 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:23.596 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:23.596 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: Reconfiguring mds.foofs.smithi022.ekdqmp (monmap changed)... 2024-03-24T04:25:23.596 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:25:23.596 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:23.597 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: Reconfiguring daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:25:23.597 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:23 smithi022 ceph-mon[66341]: pgmap v28: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-03-24T04:25:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: Reconfiguring mds.foofs.smithi022.ekdqmp (monmap changed)... 2024-03-24T04:25:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:25:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: Reconfiguring daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:25:23.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:23 smithi116 ceph-mon[53251]: pgmap v28: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-03-24T04:25:23.831 INFO:teuthology.orchestra.run.smithi022.stdout: 5 166285 11.32 MB/sec execute 225 sec latency 33.475 ms 2024-03-24T04:25:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 166685 11.32 MB/sec execute 226 sec latency 47.481 ms 2024-03-24T04:25:25.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 167080 11.34 MB/sec execute 227 sec latency 50.861 ms 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-24T04:25:26.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:26 smithi022 ceph-mon[66341]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 740 op/s 2024-03-24T04:25:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: Reconfiguring crash.smithi116 (monmap changed)... 2024-03-24T04:25:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: Reconfiguring daemon crash.smithi116 on smithi116 2024-03-24T04:25:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:26 smithi116 ceph-mon[53251]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 740 op/s 2024-03-24T04:25:26.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 167332 11.33 MB/sec execute 228 sec latency 56.968 ms 2024-03-24T04:25:27.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 167553 11.32 MB/sec execute 229 sec latency 96.290 ms 2024-03-24T04:25:27.971 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: Reconfiguring mgr.smithi116.oxktjt (monmap changed)... 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: Reconfiguring daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:25:27.972 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:27 smithi022 ceph-mon[66341]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 530 op/s 2024-03-24T04:25:28.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:28.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:28.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: Reconfiguring mgr.smithi116.oxktjt (monmap changed)... 2024-03-24T04:25:28.143 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi116.oxktjt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T04:25:28.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T04:25:28.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:28.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: Reconfiguring daemon mgr.smithi116.oxktjt on smithi116 2024-03-24T04:25:28.144 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:27 smithi116 ceph-mon[53251]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 530 op/s 2024-03-24T04:25:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 167885 11.29 MB/sec execute 230 sec latency 52.215 ms 2024-03-24T04:25:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 168146 11.27 MB/sec execute 231 sec latency 46.570 ms 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-24T04:25:29.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:29 smithi022 ceph-mon[66341]: pgmap v31: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 552 op/s 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: Reconfiguring mon.smithi116 (monmap changed)... 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: Reconfiguring daemon mon.smithi116 on smithi116 2024-03-24T04:25:30.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:29 smithi116 ceph-mon[53251]: pgmap v31: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 552 op/s 2024-03-24T04:25:30.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 168400 11.24 MB/sec execute 232 sec latency 42.685 ms 2024-03-24T04:25:31.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.4 on smithi116 2024-03-24T04:25:31.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:31 smithi022 ceph-mon[66341]: pgmap v32: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 708 op/s 2024-03-24T04:25:31.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 168678 11.24 MB/sec execute 233 sec latency 76.952 ms 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.4 on smithi116 2024-03-24T04:25:31.955 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:31 smithi116 ceph-mon[53251]: pgmap v32: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 708 op/s 2024-03-24T04:25:32.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 168995 11.20 MB/sec execute 234 sec latency 64.399 ms 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.5 on smithi116 2024-03-24T04:25:33.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: pgmap v33: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-24T04:25:33.537 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.5 on smithi116 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: pgmap v33: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-24T04:25:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 169302 11.18 MB/sec execute 235 sec latency 50.036 ms 2024-03-24T04:25:34.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 169497 11.15 MB/sec execute 236 sec latency 83.752 ms 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.6 on smithi116 2024-03-24T04:25:35.329 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:35 smithi116 ceph-mon[53251]: pgmap v34: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 664 op/s 2024-03-24T04:25:35.345 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:35.345 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:35.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T04:25:35.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:25:35.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:35.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.6 on smithi116 2024-03-24T04:25:35.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:35 smithi022 ceph-mon[66341]: pgmap v34: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 664 op/s 2024-03-24T04:25:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 169915 11.18 MB/sec execute 237 sec latency 54.662 ms 2024-03-24T04:25:36.832 INFO:teuthology.orchestra.run.smithi022.stdout: 5 170252 11.16 MB/sec execute 238 sec latency 39.044 ms 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: Reconfiguring daemon osd.7 on smithi116 2024-03-24T04:25:37.420 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:37 smithi022 ceph-mon[66341]: pgmap v35: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 371 op/s 2024-03-24T04:25:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T04:25:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:25:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: Reconfiguring daemon osd.7 on smithi116 2024-03-24T04:25:37.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:37 smithi116 ceph-mon[53251]: pgmap v35: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 371 op/s 2024-03-24T04:25:37.838 INFO:teuthology.orchestra.run.smithi022.stdout: 5 170599 11.17 MB/sec execute 239 sec latency 115.549 ms 2024-03-24T04:25:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 170836 11.16 MB/sec execute 240 sec latency 45.179 ms 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: Reconfiguring mds.foofs.smithi116.kvgpsf (monmap changed)... 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: Reconfiguring daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:25:39.737 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:39 smithi022 ceph-mon[66341]: pgmap v36: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 487 op/s 2024-03-24T04:25:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: 5 171094 11.15 MB/sec execute 241 sec latency 49.250 ms 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: Reconfiguring mds.foofs.smithi116.kvgpsf (monmap changed)... 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: Reconfiguring daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:25:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:39 smithi116 ceph-mon[53251]: pgmap v36: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 487 op/s 2024-03-24T04:25:40.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 171340 11.12 MB/sec execute 242 sec latency 72.871 ms 2024-03-24T04:25:41.010 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:25:41.530 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all mon 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi022"}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi022"}]': finished 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]': finished 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:41.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:41 smithi022 ceph-mon[66341]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-03-24T04:25:41.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 171562 11.10 MB/sec execute 243 sec latency 69.329 ms 2024-03-24T04:25:41.903 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:25:41.903 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (3m) 59s ago 12m 25.6M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:25:41.903 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (12m) 59s ago 12m 6405k - 17.2.0 e1d6a67b021e 01b192fb7d9a 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (11m) 43s ago 11m 6409k - 17.2.0 e1d6a67b021e c5e645c37b36 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (3m) 59s ago 12m 52.8M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (6m) 59s ago 6m 36.9M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (6m) 43s ago 6m 15.5M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (2m) 59s ago 13m 477M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (99s) 43s ago 11m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (79s) 59s ago 13m 31.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (50s) 43s ago 11m 21.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (6m) 59s ago 6m 265M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (6m) 43s ago 6m 87.6M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (3m) 59s ago 12m 11.7M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (2m) 43s ago 11m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:25:41.904 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (10m) 59s ago 10m 511M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (10m) 59s ago 10m 452M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (10m) 59s ago 10m 348M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (9m) 59s ago 9m 480M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (9m) 43s ago 9m 507M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (8m) 43s ago 8m 571M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (8m) 43s ago 8m 503M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (8m) 43s ago 8m 374M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:25:41.905 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (107s) 59s ago 11m 55.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:25:41.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all mon 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi022"}]: dispatch 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi022"}]': finished 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]: dispatch 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi116"}]': finished 2024-03-24T04:25:41.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:41.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:41 smithi116 ceph-mon[53251]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-03-24T04:25:42.441 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:25:42.442 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:25:42.443 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T04:25:42.443 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T04:25:42.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:25:42.443 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:25:42.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:42 smithi022 ceph-mon[66341]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:42.818 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:42 smithi022 ceph-mon[66341]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:42.819 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:42 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1680748646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 171737 11.07 MB/sec execute 244 sec latency 58.799 ms 2024-03-24T04:25:42.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:42 smithi116 ceph-mon[53251]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:42.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:42 smithi116 ceph-mon[53251]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:42.966 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:42 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1680748646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "4/23 daemons upgraded", 2024-03-24T04:25:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading crash daemons", 2024-03-24T04:25:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:25:42.982 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:25:43.736 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:25:43.823 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:43 smithi022 ceph-mon[66341]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:43.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:43 smithi022 ceph-mon[66341]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 367 op/s 2024-03-24T04:25:43.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:43.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:43.824 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 172010 11.05 MB/sec execute 245 sec latency 45.440 ms 2024-03-24T04:25:43.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:43 smithi116 ceph-mon[53251]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:25:43.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:43 smithi116 ceph-mon[53251]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 367 op/s 2024-03-24T04:25:43.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:43.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:43.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:44.834 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:44 smithi022 ceph-mon[66341]: Upgrade: Updating crash.smithi022 (1/2) 2024-03-24T04:25:44.835 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:44 smithi022 ceph-mon[66341]: Deploying daemon crash.smithi022 on smithi022 2024-03-24T04:25:44.835 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:44 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2138188273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:25:44.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 172395 11.04 MB/sec execute 246 sec latency 50.543 ms 2024-03-24T04:25:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:44 smithi116 ceph-mon[53251]: Upgrade: Updating crash.smithi022 (1/2) 2024-03-24T04:25:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:44 smithi116 ceph-mon[53251]: Deploying daemon crash.smithi022 on smithi022 2024-03-24T04:25:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:44 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2138188273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:25:45.838 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:45 smithi022 ceph-mon[66341]: pgmap v39: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-03-24T04:25:45.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 172841 11.02 MB/sec execute 247 sec latency 27.558 ms 2024-03-24T04:25:45.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:45 smithi116 ceph-mon[53251]: pgmap v39: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-03-24T04:25:46.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 173090 11.00 MB/sec execute 248 sec latency 59.699 ms 2024-03-24T04:25:47.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 173448 11.01 MB/sec execute 249 sec latency 79.095 ms 2024-03-24T04:25:48.133 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:48 smithi022 ceph-mon[66341]: pgmap v40: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 344 op/s 2024-03-24T04:25:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:48 smithi116 ceph-mon[53251]: pgmap v40: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 344 op/s 2024-03-24T04:25:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 173803 11.00 MB/sec execute 250 sec latency 56.092 ms 2024-03-24T04:25:49.150 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:49.150 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:49 smithi022 ceph-mon[66341]: pgmap v41: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 MiB/s wr, 390 op/s 2024-03-24T04:25:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:49 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:25:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:49 smithi116 ceph-mon[53251]: pgmap v41: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 MiB/s wr, 390 op/s 2024-03-24T04:25:49.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 174163 11.02 MB/sec execute 251 sec latency 42.107 ms 2024-03-24T04:25:50.681 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:50 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:50.681 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:50 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:50.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 174428 11.01 MB/sec execute 252 sec latency 45.809 ms 2024-03-24T04:25:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:50 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:50 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:51.702 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:51 smithi022 ceph-mon[66341]: pgmap v42: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.7 MiB/s wr, 439 op/s 2024-03-24T04:25:51.702 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:51.703 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:51.703 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:51.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 174764 11.01 MB/sec execute 253 sec latency 44.543 ms 2024-03-24T04:25:51.903 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:51 smithi116 ceph-mon[53251]: pgmap v42: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.7 MiB/s wr, 439 op/s 2024-03-24T04:25:51.903 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:51.904 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi116", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T04:25:51.904 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:25:52.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:52 smithi022 ceph-mon[66341]: Upgrade: Updating crash.smithi116 (2/2) 2024-03-24T04:25:52.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:52 smithi022 ceph-mon[66341]: Deploying daemon crash.smithi116 on smithi116 2024-03-24T04:25:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 175038 10.99 MB/sec execute 254 sec latency 50.314 ms 2024-03-24T04:25:52.905 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:52 smithi116 ceph-mon[53251]: Upgrade: Updating crash.smithi116 (2/2) 2024-03-24T04:25:52.905 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:52 smithi116 ceph-mon[53251]: Deploying daemon crash.smithi116 on smithi116 2024-03-24T04:25:53.732 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:53 smithi022 ceph-mon[66341]: pgmap v43: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 369 op/s 2024-03-24T04:25:53.795 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:53 smithi116 ceph-mon[53251]: pgmap v43: 97 pgs: 97 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 369 op/s 2024-03-24T04:25:53.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 175371 10.97 MB/sec execute 255 sec latency 34.312 ms 2024-03-24T04:25:54.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 175870 10.97 MB/sec execute 256 sec latency 29.262 ms 2024-03-24T04:25:55.247 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:55 smithi022 ceph-mon[66341]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 571 op/s 2024-03-24T04:25:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:55 smithi116 ceph-mon[53251]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 571 op/s 2024-03-24T04:25:55.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 176376 10.97 MB/sec execute 257 sec latency 49.452 ms 2024-03-24T04:25:56.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:56.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:56.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:25:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:25:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:25:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 176797 10.97 MB/sec execute 258 sec latency 57.051 ms 2024-03-24T04:25:57.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:57 smithi022 ceph-mon[66341]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 412 op/s 2024-03-24T04:25:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:57 smithi116 ceph-mon[53251]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 412 op/s 2024-03-24T04:25:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 177251 10.98 MB/sec execute 259 sec latency 37.361 ms 2024-03-24T04:25:58.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 177777 11.02 MB/sec execute 260 sec latency 32.417 ms 2024-03-24T04:25:59.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:25:59 smithi022 ceph-mon[66341]: pgmap v46: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 6.0 MiB/s wr, 577 op/s 2024-03-24T04:25:59.338 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:25:59 smithi116 ceph-mon[53251]: pgmap v46: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 6.0 MiB/s wr, 577 op/s 2024-03-24T04:25:59.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 178208 11.03 MB/sec execute 261 sec latency 26.881 ms 2024-03-24T04:26:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:00.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 178625 11.02 MB/sec execute 262 sec latency 31.536 ms 2024-03-24T04:26:01.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:01.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:01.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 178999 11.02 MB/sec execute 263 sec latency 39.446 ms 2024-03-24T04:26:02.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:01 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:02.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:01 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:02.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:01 smithi022 ceph-mon[66341]: pgmap v47: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-03-24T04:26:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:01 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:01 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:01 smithi116 ceph-mon[53251]: pgmap v47: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-03-24T04:26:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 179420 11.00 MB/sec execute 264 sec latency 67.877 ms 2024-03-24T04:26:03.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 179877 10.99 MB/sec execute 265 sec latency 61.494 ms 2024-03-24T04:26:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:04 smithi116 ceph-mon[53251]: pgmap v48: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-03-24T04:26:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:04.224 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:04 smithi022 ceph-mon[66341]: pgmap v48: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-03-24T04:26:04.224 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:04.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 180191 10.97 MB/sec execute 266 sec latency 44.416 ms 2024-03-24T04:26:05.239 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:05 smithi022 ceph-mon[66341]: mgrmap e43: smithi022.ijpjvk(active, since 91s), standbys: smithi116.oxktjt 2024-03-24T04:26:05.253 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:05 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:05.254 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:05 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:05.254 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:05 smithi022 ceph-mon[66341]: pgmap v49: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 745 op/s 2024-03-24T04:26:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:05 smithi116 ceph-mon[53251]: mgrmap e43: smithi022.ijpjvk(active, since 91s), standbys: smithi116.oxktjt 2024-03-24T04:26:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:05 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:05 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:05 smithi116 ceph-mon[53251]: pgmap v49: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 745 op/s 2024-03-24T04:26:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: 5 180647 11.00 MB/sec execute 267 sec latency 41.320 ms 2024-03-24T04:26:06.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:06.250 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:06.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 181080 11.00 MB/sec execute 268 sec latency 30.656 ms 2024-03-24T04:26:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 181612 11.04 MB/sec execute 269 sec latency 30.202 ms 2024-03-24T04:26:08.021 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:08.022 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:08.022 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:07 smithi022 ceph-mon[66341]: pgmap v50: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-03-24T04:26:08.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:08.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:08.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:07 smithi116 ceph-mon[53251]: pgmap v50: 97 pgs: 97 active+clean; 325 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-03-24T04:26:08.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 182002 11.03 MB/sec execute 270 sec latency 29.376 ms 2024-03-24T04:26:09.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 182482 11.03 MB/sec execute 271 sec latency 40.270 ms 2024-03-24T04:26:10.313 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:10 smithi022 ceph-mon[66341]: pgmap v51: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-03-24T04:26:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:10 smithi116 ceph-mon[53251]: pgmap v51: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-03-24T04:26:10.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 182988 11.04 MB/sec execute 272 sec latency 66.051 ms 2024-03-24T04:26:11.340 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:11 smithi022 ceph-mon[66341]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 641 op/s 2024-03-24T04:26:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:11 smithi116 ceph-mon[53251]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 641 op/s 2024-03-24T04:26:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 183299 11.01 MB/sec execute 273 sec latency 53.983 ms 2024-03-24T04:26:12.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 183612 10.99 MB/sec execute 274 sec latency 72.382 ms 2024-03-24T04:26:13.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 183915 10.98 MB/sec execute 275 sec latency 48.849 ms 2024-03-24T04:26:14.178 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.178 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.178 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: pgmap v53: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.4 MiB/s wr, 521 op/s 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.179 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all crash 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi022"}]: dispatch 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi022"}]': finished 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]: dispatch 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]': finished 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T04:26:14.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:13 smithi022 ceph-mon[66341]: Upgrade: osd.0 is safe to restart 2024-03-24T04:26:14.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: pgmap v53: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.4 MiB/s wr, 521 op/s 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all crash 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi022"}]: dispatch 2024-03-24T04:26:14.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi022"}]': finished 2024-03-24T04:26:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]: dispatch 2024-03-24T04:26:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi116"}]': finished 2024-03-24T04:26:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T04:26:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T04:26:14.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:13 smithi116 ceph-mon[53251]: Upgrade: osd.0 is safe to restart 2024-03-24T04:26:14.293 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:26:14.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 184208 10.98 MB/sec execute 276 sec latency 59.703 ms 2024-03-24T04:26:15.305 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:26:15.305 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (4m) 10s ago 13m 25.6M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:26:15.305 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (26s) 10s ago 12m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:26:15.305 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (20s) 15s ago 12m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (3m) 10s ago 12m 50.4M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (7m) 10s ago 7m 37.1M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (7m) 15s ago 7m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (3m) 10s ago 14m 496M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (2m) 15s ago 12m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (113s) 10s ago 14m 42.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (83s) 15s ago 12m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (6m) 10s ago 6m 261M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (6m) 15s ago 6m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:26:15.306 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (3m) 10s ago 12m 18.3M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (3m) 15s ago 12m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (11m) 10s ago 11m 368M 1937M 17.2.0 e1d6a67b021e 820c62bbff70 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (11m) 10s ago 11m 547M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (10m) 10s ago 10m 427M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (10m) 10s ago 10m 547M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (9m) 15s ago 9m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (9m) 15s ago 9m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (8m) 15s ago 8m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (8m) 15s ago 8m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:26:15.307 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (2m) 10s ago 11m 66.3M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:26:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 184630 10.99 MB/sec execute 277 sec latency 50.750 ms 2024-03-24T04:26:15.950 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:26:15.952 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:26:16.211 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: pgmap v54: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 755 op/s 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:16.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:16 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2683509400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: pgmap v54: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 755 op/s 2024-03-24T04:26:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T04:26:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:16 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2683509400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:26:16.574 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "6/23 daemons upgraded", 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:26:16.575 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:26:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 185163 11.03 MB/sec execute 278 sec latency 25.216 ms 2024-03-24T04:26:17.184 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:26:17.228 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:17 smithi022 ceph-mon[66341]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:17.228 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:17 smithi022 ceph-mon[66341]: Upgrade: Updating osd.0 2024-03-24T04:26:17.228 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:17 smithi022 ceph-mon[66341]: Deploying daemon osd.0 on smithi022 2024-03-24T04:26:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:17 smithi116 ceph-mon[53251]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:17 smithi116 ceph-mon[53251]: Upgrade: Updating osd.0 2024-03-24T04:26:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:17 smithi116 ceph-mon[53251]: Deploying daemon osd.0 on smithi022 2024-03-24T04:26:17.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 185536 11.03 MB/sec execute 279 sec latency 41.393 ms 2024-03-24T04:26:18.242 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:18 smithi022 ceph-mon[66341]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:18.243 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:18 smithi022 ceph-mon[66341]: pgmap v55: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 495 op/s 2024-03-24T04:26:18.243 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:18 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1613357095' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:26:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:18 smithi116 ceph-mon[53251]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:18 smithi116 ceph-mon[53251]: pgmap v55: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 495 op/s 2024-03-24T04:26:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:18 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1613357095' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:26:18.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 185808 11.00 MB/sec execute 280 sec latency 68.293 ms 2024-03-24T04:26:19.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:19.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:19 smithi022 ceph-mon[66341]: pgmap v56: 97 pgs: 97 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.5 MiB/s wr, 620 op/s 2024-03-24T04:26:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:19 smithi116 ceph-mon[53251]: pgmap v56: 97 pgs: 97 active+clean; 328 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.5 MiB/s wr, 620 op/s 2024-03-24T04:26:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 186045 10.98 MB/sec execute 281 sec latency 70.360 ms 2024-03-24T04:26:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 186445 10.98 MB/sec execute 282 sec latency 69.549 ms 2024-03-24T04:26:21.559 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:21 smithi022 systemd[1]: Stopping Ceph osd.0 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:26:21.813 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:21 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[35153]: 2024-03-24T04:26:21.752+0000 7fea07f08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:26:21.813 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:21 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[35153]: 2024-03-24T04:26:21.752+0000 7fea07f08700 -1 osd.0 72 *** Got signal Terminated *** 2024-03-24T04:26:21.813 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:21 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[35153]: 2024-03-24T04:26:21.752+0000 7fea07f08700 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:26:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 187045 10.97 MB/sec execute 283 sec latency 26.210 ms 2024-03-24T04:26:22.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:22 smithi022 ceph-mon[66341]: pgmap v57: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 588 op/s 2024-03-24T04:26:22.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:22 smithi022 ceph-mon[66341]: osd.0 marked itself down and dead 2024-03-24T04:26:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:22 smithi116 ceph-mon[53251]: pgmap v57: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 588 op/s 2024-03-24T04:26:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:22 smithi116 ceph-mon[53251]: osd.0 marked itself down and dead 2024-03-24T04:26:22.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 187324 10.97 MB/sec execute 284 sec latency 654.841 ms 2024-03-24T04:26:23.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:22 smithi022 podman[71987]: 2024-03-24 04:26:22.767809393 +0000 UTC m=+1.151410628 container died 820c62bbff70c209c4660b26e1166168f0c3a477bc10d288c8e53c599e78e445 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.buildah.version=1.19.8, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-03-24T04:26:23.286 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:23 smithi022 podman[71987]: 2024-03-24 04:26:23.150108818 +0000 UTC m=+1.533710059 container cleanup 820c62bbff70c209c4660b26e1166168f0c3a477bc10d288c8e53c599e78e445 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., distribution-scope=public, release=754, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-24T04:26:23.286 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:23 smithi022 bash[71987]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0 2024-03-24T04:26:23.286 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:26:23.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: failed to encode map e73 with expected crc 2024-03-24T04:26:23.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: osdmap e73: 8 total, 7 up, 8 in 2024-03-24T04:26:23.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: failed to encode map e73 with expected crc 2024-03-24T04:26:23.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-24T04:26:23.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:23 smithi022 ceph-mon[66341]: osdmap e74: 8 total, 7 up, 8 in 2024-03-24T04:26:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:26:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: failed to encode map e73 with expected crc 2024-03-24T04:26:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: osdmap e73: 8 total, 7 up, 8 in 2024-03-24T04:26:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: failed to encode map e73 with expected crc 2024-03-24T04:26:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-24T04:26:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:23 smithi116 ceph-mon[53251]: osdmap e74: 8 total, 7 up, 8 in 2024-03-24T04:26:23.795 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:23 smithi022 podman[72001]: 2024-03-24 04:26:23.63615879 +0000 UTC m=+0.864895801 container remove 820c62bbff70c209c4660b26e1166168f0c3a477bc10d288c8e53c599e78e445 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, io.openshift.expose-services=, architecture=x86_64, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T04:26:23.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 187716 10.96 MB/sec execute 285 sec latency 978.947 ms 2024-03-24T04:26:24.298 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:24 smithi022 podman[72089]: 2024-03-24 04:26:24.106214899 +0000 UTC m=+0.030258584 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:26:24.298 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:24 smithi022 podman[72089]: 2024-03-24 04:26:24.220649065 +0000 UTC m=+0.144692748 container create 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:26:24.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:24 smithi022 ceph-mon[66341]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:24 smithi116 ceph-mon[53251]: failed to encode map e74 with expected crc 2024-03-24T04:26:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 188293 11.00 MB/sec execute 286 sec latency 37.441 ms 2024-03-24T04:26:25.050 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:24 smithi022 podman[72089]: 2024-03-24 04:26:24.960910479 +0000 UTC m=+0.884954174 container init 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-24T04:26:25.051 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:24 smithi022 podman[72089]: 2024-03-24 04:26:24.969741109 +0000 UTC m=+0.893784795 container start 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-24T04:26:25.305 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:25 smithi022 podman[72089]: 2024-03-24 04:26:25.184334129 +0000 UTC m=+1.108377824 container attach 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T04:26:25.305 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:25 smithi022 ceph-mon[66341]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.2 MiB/s wr, 700 op/s; 216/1620 objects degraded (13.333%) 2024-03-24T04:26:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:25 smithi116 ceph-mon[53251]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.2 MiB/s wr, 700 op/s; 216/1620 objects degraded (13.333%) 2024-03-24T04:26:25.557 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:25 smithi022 podman[72089]: 2024-03-24 04:26:25.437922851 +0000 UTC m=+1.361966534 container died 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T04:26:25.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 188725 11.02 MB/sec execute 287 sec latency 32.947 ms 2024-03-24T04:26:26.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:26 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 216/1620 objects degraded (13.333%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:26 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 216/1620 objects degraded (13.333%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:26.572 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:26 smithi022 podman[72113]: 2024-03-24 04:26:26.409658238 +0000 UTC m=+0.966233792 container remove 47449193a1db6871455a6a136b1af8f2641bc2cdf15ebf9179205e3c741f4979 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:26:26.572 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:26 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T04:26:26.572 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:26 smithi022 systemd[1]: Stopped Ceph osd.0 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:26:26.572 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:26 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.0.service: Consumed 3min 16.341s CPU time. 2024-03-24T04:26:26.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:26 smithi022 systemd[1]: Starting Ceph osd.0 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:26:26.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 189219 11.02 MB/sec execute 288 sec latency 29.357 ms 2024-03-24T04:26:27.077 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:27 smithi022 podman[72240]: 2024-03-24 04:26:26.958509341 +0000 UTC m=+0.027081694 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:26:27.328 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:27 smithi022 ceph-mon[66341]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.9 MiB/s wr, 511 op/s; 216/1620 objects degraded (13.333%) 2024-03-24T04:26:27.328 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:27 smithi022 podman[72240]: 2024-03-24 04:26:27.077432093 +0000 UTC m=+0.146004438 container create f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T04:26:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:27 smithi116 ceph-mon[53251]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.9 MiB/s wr, 511 op/s; 216/1620 objects degraded (13.333%) 2024-03-24T04:26:27.579 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:27 smithi022 podman[72240]: 2024-03-24 04:26:27.45031262 +0000 UTC m=+0.518884963 container init f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T04:26:27.579 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:27 smithi022 podman[72240]: 2024-03-24 04:26:27.458418344 +0000 UTC m=+0.526990689 container start f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T04:26:27.830 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:27 smithi022 podman[72240]: 2024-03-24 04:26:27.641118676 +0000 UTC m=+0.709691023 container attach f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T04:26:27.840 INFO:teuthology.orchestra.run.smithi022.stdout: 5 189624 11.01 MB/sec execute 289 sec latency 37.431 ms 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 bash[72240]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T04:26:28.348 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T04:26:28.349 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T04:26:28.349 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T04:26:28.349 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate[72254]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T04:26:28.349 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 podman[72240]: 2024-03-24 04:26:28.12704869 +0000 UTC m=+1.195621047 container died f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T04:26:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 189881 10.98 MB/sec execute 290 sec latency 92.420 ms 2024-03-24T04:26:29.107 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:28 smithi022 podman[72381]: 2024-03-24 04:26:28.938639718 +0000 UTC m=+0.798699537 container remove f7617680bc5f7c4053ad3d0b916956b69927c9ee7d45a38308bf1cd0154e3061 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T04:26:29.360 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:29 smithi022 podman[72432]: 2024-03-24 04:26:29.084604979 +0000 UTC m=+0.021920449 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:26:29.360 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:29 smithi022 podman[72432]: 2024-03-24 04:26:29.252163874 +0000 UTC m=+0.189479315 container create 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T04:26:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 190179 10.98 MB/sec execute 291 sec latency 62.878 ms 2024-03-24T04:26:29.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:29 smithi022 podman[72432]: 2024-03-24 04:26:29.845017901 +0000 UTC m=+0.782333367 container init 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T04:26:29.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:29 smithi022 podman[72432]: 2024-03-24 04:26:29.850568791 +0000 UTC m=+0.787884242 container start 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:26:30.121 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:29 smithi022 ceph-osd[72450]: -- 172.21.15.22:0/1004994575 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1c849f1e0 con 0x55f1c8470c00 2024-03-24T04:26:30.374 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:30 smithi022 bash[72432]: 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 2024-03-24T04:26:30.375 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:30 smithi022 systemd[1]: Started Ceph osd.0 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:26:30.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:30 smithi022 ceph-mon[66341]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 6.4 MiB/s wr, 678 op/s; 225/1662 objects degraded (13.538%) 2024-03-24T04:26:30.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:30 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:26:30.436+0000 7f3e8fc677c0 -1 Falling back to public interface 2024-03-24T04:26:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:30 smithi116 ceph-mon[53251]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 6.4 MiB/s wr, 678 op/s; 225/1662 objects degraded (13.538%) 2024-03-24T04:26:30.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 190493 10.96 MB/sec execute 292 sec latency 70.054 ms 2024-03-24T04:26:31.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:31.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:31.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:31.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:31 smithi022 ceph-mon[66341]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.6 MiB/s wr, 678 op/s; 221/1698 objects degraded (13.015%) 2024-03-24T04:26:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:31 smithi116 ceph-mon[53251]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.6 MiB/s wr, 678 op/s; 221/1698 objects degraded (13.015%) 2024-03-24T04:26:31.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 190732 10.94 MB/sec execute 293 sec latency 90.433 ms 2024-03-24T04:26:32.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:32 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 221/1698 objects degraded (13.015%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:32 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 221/1698 objects degraded (13.015%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:32.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 191048 10.93 MB/sec execute 294 sec latency 43.530 ms 2024-03-24T04:26:33.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:33 smithi022 ceph-mon[66341]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 606 op/s; 221/1698 objects degraded (13.015%) 2024-03-24T04:26:33.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:33.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:33 smithi116 ceph-mon[53251]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 606 op/s; 221/1698 objects degraded (13.015%) 2024-03-24T04:26:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 191433 10.94 MB/sec execute 295 sec latency 51.228 ms 2024-03-24T04:26:34.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 191772 10.93 MB/sec execute 296 sec latency 41.835 ms 2024-03-24T04:26:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 192063 10.94 MB/sec execute 297 sec latency 44.759 ms 2024-03-24T04:26:36.141 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:35 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:26:35.956+0000 7f3e8fc677c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-24T04:26:36.141 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:36 smithi022 ceph-mon[66341]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 695 op/s; 218/1614 objects degraded (13.507%) 2024-03-24T04:26:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:36 smithi116 ceph-mon[53251]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 695 op/s; 218/1614 objects degraded (13.507%) 2024-03-24T04:26:36.841 INFO:teuthology.orchestra.run.smithi022.stdout: 5 192367 10.95 MB/sec execute 298 sec latency 49.546 ms 2024-03-24T04:26:37.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:37 smithi022 ceph-mon[66341]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 451 op/s; 218/1614 objects degraded (13.507%) 2024-03-24T04:26:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:37 smithi116 ceph-mon[53251]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 451 op/s; 218/1614 objects degraded (13.507%) 2024-03-24T04:26:37.649 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:37 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:26:37.423+0000 7f3e8fc677c0 -1 osd.0 72 log_to_monitors true 2024-03-24T04:26:37.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 192757 10.94 MB/sec execute 299 sec latency 65.444 ms 2024-03-24T04:26:38.406 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:26:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:26:38.226+0000 7f3e810bf700 -1 osd.0 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:26:38.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:38 smithi022 ceph-mon[66341]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T04:26:38.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:38 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 218/1614 objects degraded (13.507%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:38 smithi116 ceph-mon[53251]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T04:26:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:38 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 218/1614 objects degraded (13.507%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 193109 10.92 MB/sec execute 300 sec latency 66.820 ms 2024-03-24T04:26:39.423 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T04:26:39.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: failed to encode map e75 with expected crc 2024-03-24T04:26:39.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: osdmap e75: 8 total, 7 up, 8 in 2024-03-24T04:26:39.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:26:39.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: failed to encode map e75 with expected crc 2024-03-24T04:26:39.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:39 smithi022 ceph-mon[66341]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.9 MiB/s wr, 335 op/s; 223/1680 objects degraded (13.274%) 2024-03-24T04:26:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T04:26:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: failed to encode map e75 with expected crc 2024-03-24T04:26:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: osdmap e75: 8 total, 7 up, 8 in 2024-03-24T04:26:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: from='osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:26:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: failed to encode map e75 with expected crc 2024-03-24T04:26:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:39 smithi116 ceph-mon[53251]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.9 MiB/s wr, 335 op/s; 223/1680 objects degraded (13.274%) 2024-03-24T04:26:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 193290 10.90 MB/sec execute 301 sec latency 620.925 ms 2024-03-24T04:26:40.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: failed to encode map e75 with expected crc 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: failed to encode map e76 with expected crc 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456] boot 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: failed to encode map e76 with expected crc 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:40.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:40 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: failed to encode map e75 with expected crc 2024-03-24T04:26:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:26:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: failed to encode map e76 with expected crc 2024-03-24T04:26:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: osd.0 [v2:172.21.15.22:6802/1914546456,v1:172.21.15.22:6803/1914546456] boot 2024-03-24T04:26:40.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T04:26:40.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T04:26:40.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: failed to encode map e76 with expected crc 2024-03-24T04:26:40.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:40.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:40 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:40.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 193461 10.87 MB/sec execute 302 sec latency 1620.954 ms 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:41.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:41 smithi022 ceph-mon[66341]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.8 MiB/s wr, 407 op/s; 232/1782 objects degraded (13.019%) 2024-03-24T04:26:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: failed to encode map e77 with expected crc 2024-03-24T04:26:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:41.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:41.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:41 smithi116 ceph-mon[53251]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.8 MiB/s wr, 407 op/s; 232/1782 objects degraded (13.019%) 2024-03-24T04:26:41.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 193593 10.85 MB/sec execute 303 sec latency 2315.934 ms 2024-03-24T04:26:42.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:42 smithi022 ceph-mon[66341]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:42 smithi022 ceph-mon[66341]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T04:26:42.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:42 smithi022 ceph-mon[66341]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:42 smithi022 ceph-mon[66341]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:42 smithi022 ceph-mon[66341]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:42 smithi116 ceph-mon[53251]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:42 smithi116 ceph-mon[53251]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T04:26:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:42 smithi116 ceph-mon[53251]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:42 smithi116 ceph-mon[53251]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:42 smithi116 ceph-mon[53251]: failed to encode map e78 with expected crc 2024-03-24T04:26:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 193912 10.84 MB/sec execute 304 sec latency 55.256 ms 2024-03-24T04:26:43.234 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T04:26:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 232/1782 objects degraded (13.019%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:43 smithi116 ceph-mon[53251]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 490 op/s; 232/1782 objects degraded (13.019%) 2024-03-24T04:26:43.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T04:26:43.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: failed to encode map e79 with expected crc 2024-03-24T04:26:43.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 232/1782 objects degraded (13.019%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:43.487 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:43 smithi022 ceph-mon[66341]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 490 op/s; 232/1782 objects degraded (13.019%) 2024-03-24T04:26:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 194374 10.84 MB/sec execute 305 sec latency 56.353 ms 2024-03-24T04:26:44.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:44 smithi022 ceph-mon[66341]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:44 smithi022 ceph-mon[66341]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T04:26:44.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:44 smithi022 ceph-mon[66341]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:44 smithi022 ceph-mon[66341]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.258 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:44 smithi022 ceph-mon[66341]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:44 smithi116 ceph-mon[53251]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:44 smithi116 ceph-mon[53251]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T04:26:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:44 smithi116 ceph-mon[53251]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:44 smithi116 ceph-mon[53251]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:44 smithi116 ceph-mon[53251]: failed to encode map e80 with expected crc 2024-03-24T04:26:44.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 194869 10.84 MB/sec execute 306 sec latency 406.213 ms 2024-03-24T04:26:45.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:45 smithi022 ceph-mon[66341]: failed to encode map e81 with expected crc 2024-03-24T04:26:45.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:45 smithi022 ceph-mon[66341]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T04:26:45.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:45 smithi022 ceph-mon[66341]: failed to encode map e81 with expected crc 2024-03-24T04:26:45.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:45 smithi022 ceph-mon[66341]: pgmap v78: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 493 op/s; 68/1590 objects degraded (4.277%); 3.0 MiB/s, 48 objects/s recovering 2024-03-24T04:26:45.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:45 smithi116 ceph-mon[53251]: failed to encode map e81 with expected crc 2024-03-24T04:26:45.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:45 smithi116 ceph-mon[53251]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T04:26:45.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:45 smithi116 ceph-mon[53251]: failed to encode map e81 with expected crc 2024-03-24T04:26:45.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:45 smithi116 ceph-mon[53251]: pgmap v78: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 493 op/s; 68/1590 objects degraded (4.277%); 3.0 MiB/s, 48 objects/s recovering 2024-03-24T04:26:45.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 195436 10.88 MB/sec execute 307 sec latency 811.046 ms 2024-03-24T04:26:46.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T04:26:46.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.318 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.319 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:26:46.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: failed to encode map e82 with expected crc 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:46.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:26:46.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 195844 10.89 MB/sec execute 308 sec latency 599.020 ms 2024-03-24T04:26:47.331 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:47 smithi022 ceph-mon[66341]: pgmap v81: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 493 op/s; 68/1590 objects degraded (4.277%); 3.0 MiB/s, 48 objects/s recovering 2024-03-24T04:26:47.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:26:47.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:26:47.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T04:26:47.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: failed to encode map e83 with expected crc 2024-03-24T04:26:47.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:47 smithi116 ceph-mon[53251]: pgmap v81: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 493 op/s; 68/1590 objects degraded (4.277%); 3.0 MiB/s, 48 objects/s recovering 2024-03-24T04:26:47.767 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:26:47.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 196280 10.89 MB/sec execute 309 sec latency 1025.327 ms 2024-03-24T04:26:48.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 68/1590 objects degraded (4.277%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:48.343 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: failed to encode map e84 with expected crc 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 68/1590 objects degraded (4.277%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:48.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (4m) 9s ago 13m 24.7M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (59s) 9s ago 13m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (54s) 49s ago 12m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (4m) 9s ago 13m 51.3M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (7m) 9s ago 7m 37.8M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (7m) 49s ago 7m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:26:48.831 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (3m) 9s ago 14m 499M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (2m) 49s ago 12m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (2m) 9s ago 15m 43.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (117s) 49s ago 12m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (7m) 9s ago 7m 264M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (7m) 49s ago 7m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (4m) 9s ago 13m 18.4M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (3m) 49s ago 12m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (19s) 9s ago 12m 12.3M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (11m) 9s ago 11m 580M 1937M 17.2.0 e1d6a67b021e c9f53c659383 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (11m) 9s ago 11m 481M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:26:48.832 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (10m) 9s ago 10m 595M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:26:48.833 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (10m) 49s ago 10m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:26:48.833 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (9m) 49s ago 9m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:26:48.833 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (9m) 49s ago 9m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:26:48.833 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (9m) 49s ago 9m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:26:48.833 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (2m) 9s ago 12m 66.3M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:26:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 196631 10.88 MB/sec execute 310 sec latency 50.521 ms 2024-03-24T04:26:49.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:49 smithi022 ceph-mon[66341]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:49.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:49 smithi022 ceph-mon[66341]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:49.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:49 smithi022 ceph-mon[66341]: pgmap v83: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 323 op/s; 25/1701 objects degraded (1.470%); 0 B/s, 8 objects/s recovering 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:26:49.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:26:49.569 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:26:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:49 smithi116 ceph-mon[53251]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:49 smithi116 ceph-mon[53251]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:49 smithi116 ceph-mon[53251]: pgmap v83: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 323 op/s; 25/1701 objects degraded (1.470%); 0 B/s, 8 objects/s recovering 2024-03-24T04:26:49.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 197017 10.86 MB/sec execute 311 sec latency 78.846 ms 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:26:50.167 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "7/23 daemons upgraded", 2024-03-24T04:26:50.168 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:26:50.168 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:26:50.168 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:26:50.366 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:50 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1262033965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:50 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1262033965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:26:50.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 197390 10.85 MB/sec execute 312 sec latency 44.382 ms 2024-03-24T04:26:50.842 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN Degraded data redundancy: 25/1701 objects degraded (1.470%), 1 pg degraded 2024-03-24T04:26:50.843 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 25/1701 objects degraded (1.470%), 1 pg degraded 2024-03-24T04:26:50.843 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.18 is active+recovery_wait+undersized+degraded+remapped, acting [3,1] 2024-03-24T04:26:51.374 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:51 smithi022 ceph-mon[66341]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:51.374 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:51 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3046559995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:26:51.374 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:51 smithi022 ceph-mon[66341]: pgmap v84: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 6.1 MiB/s wr, 498 op/s; 0 B/s, 12 objects/s recovering 2024-03-24T04:26:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:51 smithi116 ceph-mon[53251]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:26:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:51 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3046559995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:26:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:51 smithi116 ceph-mon[53251]: pgmap v84: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 6.1 MiB/s wr, 498 op/s; 0 B/s, 12 objects/s recovering 2024-03-24T04:26:51.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 197819 10.86 MB/sec execute 313 sec latency 36.629 ms 2024-03-24T04:26:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:52 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/1701 objects degraded (1.470%), 1 pg degraded) 2024-03-24T04:26:52.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:52 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:26:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:52 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/1701 objects degraded (1.470%), 1 pg degraded) 2024-03-24T04:26:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:52 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:26:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 198317 10.86 MB/sec execute 314 sec latency 53.420 ms 2024-03-24T04:26:53.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:53 smithi022 ceph-mon[66341]: pgmap v85: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.7 MiB/s wr, 381 op/s; 0 B/s, 9 objects/s recovering 2024-03-24T04:26:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:53 smithi116 ceph-mon[53251]: pgmap v85: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.7 MiB/s wr, 381 op/s; 0 B/s, 9 objects/s recovering 2024-03-24T04:26:53.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 198738 10.87 MB/sec execute 315 sec latency 35.044 ms 2024-03-24T04:26:54.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 199135 10.89 MB/sec execute 316 sec latency 37.690 ms 2024-03-24T04:26:55.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 199489 10.89 MB/sec execute 317 sec latency 40.195 ms 2024-03-24T04:26:56.205 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:56 smithi022 ceph-mon[66341]: pgmap v86: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 711 op/s; 0 B/s, 8 objects/s recovering 2024-03-24T04:26:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:56 smithi116 ceph-mon[53251]: pgmap v86: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 711 op/s; 0 B/s, 8 objects/s recovering 2024-03-24T04:26:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: 5 199811 10.88 MB/sec execute 318 sec latency 44.653 ms 2024-03-24T04:26:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:57 smithi022 ceph-mon[66341]: pgmap v87: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 629 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T04:26:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:57 smithi116 ceph-mon[53251]: pgmap v87: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.5 MiB/s wr, 629 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T04:26:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 200027 10.86 MB/sec execute 319 sec latency 55.089 ms 2024-03-24T04:26:58.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 200301 10.85 MB/sec execute 320 sec latency 124.331 ms 2024-03-24T04:26:59.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:59 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:59.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:26:59 smithi022 ceph-mon[66341]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 7.3 MiB/s wr, 703 op/s; 0 B/s, 6 objects/s recovering 2024-03-24T04:26:59.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:59 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:26:59.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:26:59 smithi116 ceph-mon[53251]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 7.3 MiB/s wr, 703 op/s; 0 B/s, 6 objects/s recovering 2024-03-24T04:26:59.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 200575 10.82 MB/sec execute 321 sec latency 71.177 ms 2024-03-24T04:27:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 200865 10.80 MB/sec execute 322 sec latency 61.310 ms 2024-03-24T04:27:01.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 201527 10.84 MB/sec execute 323 sec latency 29.874 ms 2024-03-24T04:27:02.137 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:02 smithi022 ceph-mon[66341]: pgmap v89: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 587 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:27:02.137 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:02 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:27:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:02 smithi116 ceph-mon[53251]: pgmap v89: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 587 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:27:02.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:02 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:27:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 202240 10.86 MB/sec execute 324 sec latency 23.465 ms 2024-03-24T04:27:03.145 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:03 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:27:03.145 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:03 smithi022 ceph-mon[66341]: Upgrade: osd.1 is safe to restart 2024-03-24T04:27:03.461 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:03 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T04:27:03.461 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:03 smithi116 ceph-mon[53251]: Upgrade: osd.1 is safe to restart 2024-03-24T04:27:03.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 202724 10.88 MB/sec execute 325 sec latency 27.752 ms 2024-03-24T04:27:04.148 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: pgmap v90: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 469 op/s 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: Upgrade: Updating osd.1 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:04.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:04 smithi022 ceph-mon[66341]: Deploying daemon osd.1 on smithi022 2024-03-24T04:27:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: pgmap v90: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 469 op/s 2024-03-24T04:27:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: Upgrade: Updating osd.1 2024-03-24T04:27:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T04:27:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:04 smithi116 ceph-mon[53251]: Deploying daemon osd.1 on smithi022 2024-03-24T04:27:04.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 203299 10.90 MB/sec execute 326 sec latency 46.838 ms 2024-03-24T04:27:05.157 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:05 smithi022 ceph-mon[66341]: pgmap v91: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 760 op/s 2024-03-24T04:27:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:05 smithi116 ceph-mon[53251]: pgmap v91: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 760 op/s 2024-03-24T04:27:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 203620 10.90 MB/sec execute 327 sec latency 45.899 ms 2024-03-24T04:27:06.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 203985 10.88 MB/sec execute 328 sec latency 46.177 ms 2024-03-24T04:27:07.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 204343 10.86 MB/sec execute 329 sec latency 53.456 ms 2024-03-24T04:27:08.189 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:08 smithi022 ceph-mon[66341]: pgmap v92: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 485 op/s 2024-03-24T04:27:08.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:08 smithi116 ceph-mon[53251]: pgmap v92: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 485 op/s 2024-03-24T04:27:08.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 204759 10.87 MB/sec execute 330 sec latency 49.165 ms 2024-03-24T04:27:09.198 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:09 smithi022 ceph-mon[66341]: pgmap v93: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.2 MiB/s wr, 537 op/s 2024-03-24T04:27:09.457 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:09 smithi022 systemd[1]: Stopping Ceph osd.1 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:27:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:09 smithi116 ceph-mon[53251]: pgmap v93: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.2 MiB/s wr, 537 op/s 2024-03-24T04:27:09.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 205268 10.88 MB/sec execute 331 sec latency 37.593 ms 2024-03-24T04:27:09.967 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:09 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[38016]: 2024-03-24T04:27:09.794+0000 7faa761ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:27:09.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:09 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[38016]: 2024-03-24T04:27:09.794+0000 7faa761ee700 -1 osd.1 84 *** Got signal Terminated *** 2024-03-24T04:27:09.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:09 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[38016]: 2024-03-24T04:27:09.794+0000 7faa761ee700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:27:10.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:10 smithi022 ceph-mon[66341]: osd.1 marked itself down and dead 2024-03-24T04:27:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:10 smithi116 ceph-mon[53251]: osd.1 marked itself down and dead 2024-03-24T04:27:10.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 205579 10.87 MB/sec execute 332 sec latency 637.223 ms 2024-03-24T04:27:11.120 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:10 smithi022 podman[74561]: 2024-03-24 04:27:10.829910527 +0000 UTC m=+1.208330506 container died c9f53c659383c488223764d744116cee4cb654bedb945e5921394b422f765fd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T04:27:11.372 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:11 smithi022 podman[74561]: 2024-03-24 04:27:11.220066738 +0000 UTC m=+1.598486731 container cleanup c9f53c659383c488223764d744116cee4cb654bedb945e5921394b422f765fd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, release=754, io.openshift.tags=base centos centos-stream) 2024-03-24T04:27:11.372 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:11 smithi022 bash[74561]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: osdmap e85: 8 total, 7 up, 8 in 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:11 smithi022 ceph-mon[66341]: pgmap v95: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.5 MiB/s wr, 664 op/s 2024-03-24T04:27:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:27:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: osdmap e85: 8 total, 7 up, 8 in 2024-03-24T04:27:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: failed to encode map e85 with expected crc 2024-03-24T04:27:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:11 smithi116 ceph-mon[53251]: pgmap v95: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.5 MiB/s wr, 664 op/s 2024-03-24T04:27:11.623 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:11 smithi022 podman[74577]: 2024-03-24 04:27:11.565265396 +0000 UTC m=+0.738126864 container remove c9f53c659383c488223764d744116cee4cb654bedb945e5921394b422f765fd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, architecture=x86_64, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8) 2024-03-24T04:27:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 205804 10.86 MB/sec execute 333 sec latency 1012.244 ms 2024-03-24T04:27:12.127 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:11.942339168 +0000 UTC m=+0.040896347 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:27:12.127 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:12.074927443 +0000 UTC m=+0.173484610 container create 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: osdmap e86: 8 total, 7 up, 8 in 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:12 smithi022 ceph-mon[66341]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.384 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:12.375511524 +0000 UTC m=+0.474068705 container init 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T04:27:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: osdmap e86: 8 total, 7 up, 8 in 2024-03-24T04:27:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:12 smithi116 ceph-mon[53251]: failed to encode map e86 with expected crc 2024-03-24T04:27:12.635 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:12.385954772 +0000 UTC m=+0.484511945 container start 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T04:27:12.635 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:12.570578373 +0000 UTC m=+0.669135549 container attach 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T04:27:12.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 206096 10.86 MB/sec execute 334 sec latency 33.113 ms 2024-03-24T04:27:12.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:12 smithi022 podman[74662]: 2024-03-24 04:27:12.860747898 +0000 UTC m=+0.959305073 container died 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T04:27:13.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:13 smithi022 ceph-mon[66341]: pgmap v97: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.6 MiB/s wr, 393 op/s 2024-03-24T04:27:13.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:13 smithi116 ceph-mon[53251]: pgmap v97: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.6 MiB/s wr, 393 op/s 2024-03-24T04:27:13.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 206472 10.85 MB/sec execute 335 sec latency 36.249 ms 2024-03-24T04:27:13.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:13 smithi022 podman[74689]: 2024-03-24 04:27:13.796734269 +0000 UTC m=+0.925234108 container remove 798b5a71657157c527eedd6d23f5e7cdbf9469732bf30d8e79d3e5999fc431ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T04:27:13.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:13 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T04:27:13.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:13 smithi022 systemd[1]: Stopped Ceph osd.1 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:27:13.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:13 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.1.service: Consumed 3min 44.410s CPU time. 2024-03-24T04:27:14.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 systemd[1]: Starting Ceph osd.1 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:27:14.422 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 podman[74783]: 2024-03-24 04:27:14.308929347 +0000 UTC m=+0.025890362 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:27:14.674 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 podman[74783]: 2024-03-24 04:27:14.444762204 +0000 UTC m=+0.161723205 container create 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T04:27:14.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 206793 10.85 MB/sec execute 336 sec latency 48.222 ms 2024-03-24T04:27:14.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 podman[74783]: 2024-03-24 04:27:14.810194504 +0000 UTC m=+0.527155509 container init 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T04:27:14.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 podman[74783]: 2024-03-24 04:27:14.818575074 +0000 UTC m=+0.535536069 container start 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T04:27:15.189 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:14 smithi022 podman[74783]: 2024-03-24 04:27:14.982931885 +0000 UTC m=+0.699892883 container attach 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T04:27:15.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 bash[74783]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T04:27:15.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate[74797]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T04:27:15.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-mon[66341]: pgmap v98: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 678 op/s; 193/1668 objects degraded (11.571%) 2024-03-24T04:27:15.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:15 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 193/1668 objects degraded (11.571%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:15 smithi116 ceph-mon[53251]: pgmap v98: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 678 op/s; 193/1668 objects degraded (11.571%) 2024-03-24T04:27:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:15 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 193/1668 objects degraded (11.571%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:15.691 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:15 smithi022 podman[74783]: 2024-03-24 04:27:15.443132032 +0000 UTC m=+1.160093026 container died 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T04:27:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 207094 10.85 MB/sec execute 337 sec latency 38.443 ms 2024-03-24T04:27:16.701 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:16 smithi022 podman[74929]: 2024-03-24 04:27:16.560776021 +0000 UTC m=+1.105511097 container remove 2c35cde8fd2f8e29f76ceac7e094e76584e5ceada77c3c7b2040f27f22e4ac1e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True) 2024-03-24T04:27:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 207484 10.84 MB/sec execute 338 sec latency 39.412 ms 2024-03-24T04:27:16.956 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:16 smithi022 podman[74985]: 2024-03-24 04:27:16.724061589 +0000 UTC m=+0.032178466 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:27:16.956 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:16 smithi022 podman[74985]: 2024-03-24 04:27:16.85366467 +0000 UTC m=+0.161781524 container create 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T04:27:17.466 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 podman[74985]: 2024-03-24 04:27:17.381626923 +0000 UTC m=+0.689743794 container init 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, ceph=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T04:27:17.466 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 podman[74985]: 2024-03-24 04:27:17.389369911 +0000 UTC m=+0.697486769 container start 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T04:27:17.466 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 ceph-osd[75003]: -- 172.21.15.22:0/2823801836 <== mon.0 v2:172.21.15.22:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f91133d1e0 con 0x55f91130ec00 2024-03-24T04:27:17.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 207794 10.82 MB/sec execute 339 sec latency 55.030 ms 2024-03-24T04:27:18.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 bash[74985]: 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 2024-03-24T04:27:18.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 systemd[1]: Started Ceph osd.1 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:27:18.048 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:17 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:27:17.975+0000 7f1be00ab7c0 -1 Falling back to public interface 2024-03-24T04:27:18.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:18 smithi022 ceph-mon[66341]: pgmap v99: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 599 op/s; 193/1668 objects degraded (11.571%) 2024-03-24T04:27:18.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:18.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:18.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:27:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:18 smithi116 ceph-mon[53251]: pgmap v99: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 599 op/s; 193/1668 objects degraded (11.571%) 2024-03-24T04:27:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:27:18.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 207978 10.80 MB/sec execute 340 sec latency 57.984 ms 2024-03-24T04:27:19.340 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:19.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:19.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:19 smithi022 ceph-mon[66341]: pgmap v100: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 312 op/s; 190/1686 objects degraded (11.269%) 2024-03-24T04:27:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:19 smithi116 ceph-mon[53251]: pgmap v100: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 312 op/s; 190/1686 objects degraded (11.269%) 2024-03-24T04:27:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 208154 10.78 MB/sec execute 341 sec latency 57.805 ms 2024-03-24T04:27:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 208471 10.79 MB/sec execute 342 sec latency 80.236 ms 2024-03-24T04:27:21.323 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:27:21.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:21 smithi022 ceph-mon[66341]: pgmap v101: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 466 op/s; 186/1632 objects degraded (11.397%) 2024-03-24T04:27:21.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:21 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 186/1632 objects degraded (11.397%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:21 smithi116 ceph-mon[53251]: pgmap v101: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 466 op/s; 186/1632 objects degraded (11.397%) 2024-03-24T04:27:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:21 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 186/1632 objects degraded (11.397%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 208773 10.77 MB/sec execute 343 sec latency 77.528 ms 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 42s ago 14m 24.7M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (93s) 42s ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (87s) 82s ago 13m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (4m) 42s ago 13m 51.3M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (8m) 42s ago 8m 37.8M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:27:22.248 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (8m) 82s ago 8m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (4m) 42s ago 15m 499M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (3m) 82s ago 13m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (3m) 42s ago 15m 43.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (2m) 82s ago 13m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (7m) 42s ago 7m 264M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (7m) 82s ago 7m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (4m) 42s ago 13m 18.4M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (4m) 82s ago 13m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (53s) 42s ago 12m 12.3M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 starting - - - 1937M 2024-03-24T04:27:22.249 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (11m) 42s ago 11m 481M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (11m) 42s ago 11m 595M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (10m) 82s ago 10m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (10m) 82s ago 10m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (10m) 82s ago 10m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (9m) 82s ago 9m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:27:22.250 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (3m) 42s ago 12m 66.3M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:27:22.432 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:22 smithi022 ceph-mon[66341]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:22.432 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:22 smithi022 ceph-mon[66341]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:22 smithi116 ceph-mon[53251]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:22 smithi116 ceph-mon[53251]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:22.843 INFO:teuthology.orchestra.run.smithi022.stdout: 5 209145 10.77 MB/sec execute 344 sec latency 39.779 ms 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:27:22.852 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:27:22.853 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:27:23.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:23 smithi022 ceph-mon[66341]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:23.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:23 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/119283342' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:23.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:23 smithi022 ceph-mon[66341]: pgmap v102: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 186/1632 objects degraded (11.397%) 2024-03-24T04:27:23.444 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:27:23.444 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "7/23 daemons upgraded", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:27:23.445 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:27:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:23 smithi116 ceph-mon[53251]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:23 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/119283342' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:23 smithi116 ceph-mon[53251]: pgmap v102: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 186/1632 objects degraded (11.397%) 2024-03-24T04:27:23.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:23 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:27:23.493+0000 7f1be00ab7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-24T04:27:23.844 INFO:teuthology.orchestra.run.smithi022.stdout: 5 209370 10.76 MB/sec execute 345 sec latency 47.310 ms 2024-03-24T04:27:24.108 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 186/1632 objects degraded (11.397%), 24 pgs degraded 2024-03-24T04:27:24.108 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T04:27:24.108 INFO:teuthology.orchestra.run.smithi022.stdout: osd.1 (root=default,host=smithi022) is down 2024-03-24T04:27:24.108 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 186/1632 objects degraded (11.397%), 24 pgs degraded 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-03-24T04:27:24.109 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.a is active+undersized+degraded, acting [6,7] 2024-03-24T04:27:24.110 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-03-24T04:27:24.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:24 smithi022 ceph-mon[66341]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:24.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:24 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2107724457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:27:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:24 smithi116 ceph-mon[53251]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:24 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2107724457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:27:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: 5 209614 10.76 MB/sec execute 346 sec latency 45.806 ms 2024-03-24T04:27:25.017 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:24 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:27:24.879+0000 7f1be00ab7c0 -1 osd.1 84 log_to_monitors true 2024-03-24T04:27:25.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:25 smithi022 ceph-mon[66341]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T04:27:25.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:25 smithi022 ceph-mon[66341]: pgmap v103: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 551 op/s; 180/1623 objects degraded (11.091%) 2024-03-24T04:27:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:25 smithi116 ceph-mon[53251]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T04:27:25.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:25 smithi116 ceph-mon[53251]: pgmap v103: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 551 op/s; 180/1623 objects degraded (11.091%) 2024-03-24T04:27:25.796 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:27:25 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:27:25.543+0000 7f1bcd4fb700 -1 osd.1 84 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:27:25.850 INFO:teuthology.orchestra.run.smithi022.stdout: 5 209938 10.75 MB/sec execute 347 sec latency 62.190 ms 2024-03-24T04:27:26.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T04:27:26.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: failed to encode map e87 with expected crc 2024-03-24T04:27:26.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: osdmap e87: 8 total, 7 up, 8 in 2024-03-24T04:27:26.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:27:26.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: failed to encode map e87 with expected crc 2024-03-24T04:27:26.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:26 smithi022 ceph-mon[66341]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' 2024-03-24T04:27:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T04:27:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: failed to encode map e87 with expected crc 2024-03-24T04:27:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: osdmap e87: 8 total, 7 up, 8 in 2024-03-24T04:27:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:27:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: failed to encode map e87 with expected crc 2024-03-24T04:27:26.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:26 smithi116 ceph-mon[53251]: from='osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386]' entity='osd.1' 2024-03-24T04:27:26.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 210254 10.74 MB/sec execute 348 sec latency 286.906 ms 2024-03-24T04:27:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:27:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: failed to encode map e88 with expected crc 2024-03-24T04:27:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386] boot 2024-03-24T04:27:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T04:27:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:27:27.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:27 smithi116 ceph-mon[53251]: pgmap v106: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 480 op/s; 180/1623 objects degraded (11.091%) 2024-03-24T04:27:27.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:27:27.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: failed to encode map e88 with expected crc 2024-03-24T04:27:27.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: osd.1 [v2:172.21.15.22:6810/1657055386,v1:172.21.15.22:6811/1657055386] boot 2024-03-24T04:27:27.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T04:27:27.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T04:27:27.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:27 smithi022 ceph-mon[66341]: pgmap v106: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 480 op/s; 180/1623 objects degraded (11.091%) 2024-03-24T04:27:27.850 INFO:teuthology.orchestra.run.smithi022.stdout: 5 210351 10.72 MB/sec execute 349 sec latency 1061.335 ms 2024-03-24T04:27:28.389 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.390 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:28 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 180/1623 objects degraded (11.091%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:28.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:28 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 180/1623 objects degraded (11.091%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:28.853 INFO:teuthology.orchestra.run.smithi022.stdout: 5 210497 10.71 MB/sec execute 350 sec latency 1118.674 ms 2024-03-24T04:27:29.438 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e89 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:29.439 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:29 smithi022 ceph-mon[66341]: pgmap v109: 97 pgs: 69 active+clean, 8 active+undersized, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+undersized+remapped; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.9 MiB/s wr, 97 op/s; 146/1626 objects degraded (8.979%); 64 KiB/s, 7 objects/s recovering 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e89 with expected crc 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:29.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:29 smithi116 ceph-mon[53251]: pgmap v109: 97 pgs: 69 active+clean, 8 active+undersized, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+undersized+remapped; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.9 MiB/s wr, 97 op/s; 146/1626 objects degraded (8.979%); 64 KiB/s, 7 objects/s recovering 2024-03-24T04:27:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 210750 10.71 MB/sec execute 351 sec latency 1137.989 ms 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: failed to encode map e90 with expected crc 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:30 smithi022 ceph-mon[66341]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: failed to encode map e90 with expected crc 2024-03-24T04:27:30.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T04:27:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:30 smithi116 ceph-mon[53251]: failed to encode map e91 with expected crc 2024-03-24T04:27:30.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 211043 10.68 MB/sec execute 352 sec latency 871.582 ms 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e91 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.517 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:31 smithi022 ceph-mon[66341]: pgmap v112: 97 pgs: 1 peering, 91 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 426 op/s; 54/1596 objects degraded (3.383%); 64 KiB/s, 64 keys/s, 33 objects/s recovering 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e91 with expected crc 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:31.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:31 smithi116 ceph-mon[53251]: pgmap v112: 97 pgs: 1 peering, 91 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 426 op/s; 54/1596 objects degraded (3.383%); 64 KiB/s, 64 keys/s, 33 objects/s recovering 2024-03-24T04:27:31.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 211371 10.66 MB/sec execute 353 sec latency 958.550 ms 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e92 with expected crc 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.560 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.560 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.560 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:32 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e92 with expected crc 2024-03-24T04:27:32.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:32 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:32.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 211722 10.66 MB/sec execute 354 sec latency 956.871 ms 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: failed to encode map e93 with expected crc 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.571 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 54/1596 objects degraded (3.383%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:33.572 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: pgmap v115: 97 pgs: 1 peering, 91 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.4 MiB/s wr, 328 op/s; 54/1596 objects degraded (3.383%); 22 B/s, 64 keys/s, 26 objects/s recovering 2024-03-24T04:27:33.572 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:33.572 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: failed to encode map e93 with expected crc 2024-03-24T04:27:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T04:27:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: failed to encode map e94 with expected crc 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 54/1596 objects degraded (3.383%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: pgmap v115: 97 pgs: 1 peering, 91 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.4 MiB/s wr, 328 op/s; 54/1596 objects degraded (3.383%); 22 B/s, 64 keys/s, 26 objects/s recovering 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 212199 10.66 MB/sec execute 355 sec latency 54.763 ms 2024-03-24T04:27:34.578 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:34 smithi022 ceph-mon[66341]: failed to encode map e94 with expected crc 2024-03-24T04:27:34.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:34 smithi116 ceph-mon[53251]: failed to encode map e94 with expected crc 2024-03-24T04:27:34.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 212630 10.66 MB/sec execute 356 sec latency 36.199 ms 2024-03-24T04:27:35.602 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.602 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.602 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:35.602 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:27:35.603 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:35 smithi022 ceph-mon[66341]: pgmap v116: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.4 MiB/s wr, 466 op/s; 21/1578 objects degraded (1.331%); 0 B/s, 48 keys/s, 18 objects/s recovering 2024-03-24T04:27:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:27:35.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:35 smithi116 ceph-mon[53251]: pgmap v116: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.4 MiB/s wr, 466 op/s; 21/1578 objects degraded (1.331%); 0 B/s, 48 keys/s, 18 objects/s recovering 2024-03-24T04:27:35.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 213096 10.68 MB/sec execute 357 sec latency 35.263 ms 2024-03-24T04:27:36.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 213543 10.71 MB/sec execute 358 sec latency 30.224 ms 2024-03-24T04:27:37.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:37 smithi022 ceph-mon[66341]: pgmap v117: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.2 MiB/s wr, 403 op/s; 669 KiB/s, 11 objects/s recovering 2024-03-24T04:27:37.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:37 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 1 pg degraded) 2024-03-24T04:27:37.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:37 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:27:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:37 smithi116 ceph-mon[53251]: pgmap v117: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.2 MiB/s wr, 403 op/s; 669 KiB/s, 11 objects/s recovering 2024-03-24T04:27:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:37 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 1 pg degraded) 2024-03-24T04:27:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:37 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:27:37.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 213964 10.70 MB/sec execute 359 sec latency 41.400 ms 2024-03-24T04:27:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 214339 10.69 MB/sec execute 360 sec latency 31.866 ms 2024-03-24T04:27:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 214776 10.69 MB/sec execute 361 sec latency 47.032 ms 2024-03-24T04:27:40.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:40 smithi022 ceph-mon[66341]: pgmap v118: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 502 op/s; 532 KiB/s, 9 objects/s recovering 2024-03-24T04:27:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:40 smithi116 ceph-mon[53251]: pgmap v118: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 502 op/s; 532 KiB/s, 9 objects/s recovering 2024-03-24T04:27:40.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 215264 10.69 MB/sec execute 362 sec latency 64.446 ms 2024-03-24T04:27:41.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 215782 10.69 MB/sec execute 363 sec latency 36.893 ms 2024-03-24T04:27:42.290 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:42 smithi022 ceph-mon[66341]: pgmap v119: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.4 MiB/s wr, 664 op/s; 455 KiB/s, 8 objects/s recovering 2024-03-24T04:27:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:42 smithi116 ceph-mon[53251]: pgmap v119: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.4 MiB/s wr, 664 op/s; 455 KiB/s, 8 objects/s recovering 2024-03-24T04:27:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 216226 10.69 MB/sec execute 364 sec latency 32.345 ms 2024-03-24T04:27:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 216668 10.71 MB/sec execute 365 sec latency 32.697 ms 2024-03-24T04:27:44.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:44 smithi022 ceph-mon[66341]: pgmap v120: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.8 MiB/s wr, 600 op/s; 411 KiB/s, 7 objects/s recovering 2024-03-24T04:27:44.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:44 smithi116 ceph-mon[53251]: pgmap v120: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.8 MiB/s wr, 600 op/s; 411 KiB/s, 7 objects/s recovering 2024-03-24T04:27:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:44 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:44.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 217081 10.73 MB/sec execute 366 sec latency 31.150 ms 2024-03-24T04:27:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 217441 10.72 MB/sec execute 367 sec latency 34.901 ms 2024-03-24T04:27:46.345 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:46 smithi022 ceph-mon[66341]: pgmap v121: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 696 op/s; 355 KiB/s, 6 objects/s recovering 2024-03-24T04:27:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:46 smithi116 ceph-mon[53251]: pgmap v121: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 696 op/s; 355 KiB/s, 6 objects/s recovering 2024-03-24T04:27:46.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 217724 10.71 MB/sec execute 368 sec latency 35.712 ms 2024-03-24T04:27:47.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:47 smithi022 ceph-mon[66341]: pgmap v122: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.9 MiB/s wr, 546 op/s; 341 KiB/s, 1 objects/s recovering 2024-03-24T04:27:47.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:47 smithi116 ceph-mon[53251]: pgmap v122: 97 pgs: 97 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.9 MiB/s wr, 546 op/s; 341 KiB/s, 1 objects/s recovering 2024-03-24T04:27:47.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 218138 10.70 MB/sec execute 369 sec latency 46.574 ms 2024-03-24T04:27:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: 5 218603 10.69 MB/sec execute 370 sec latency 54.340 ms 2024-03-24T04:27:49.149 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:49.150 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:49 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:49 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:27:49.851 INFO:teuthology.orchestra.run.smithi022.stdout: 5 219047 10.69 MB/sec execute 371 sec latency 60.446 ms 2024-03-24T04:27:50.183 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:50 smithi022 ceph-mon[66341]: pgmap v123: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 7.1 MiB/s wr, 662 op/s 2024-03-24T04:27:50.183 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:50 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:50 smithi116 ceph-mon[53251]: pgmap v123: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 7.1 MiB/s wr, 662 op/s 2024-03-24T04:27:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:50 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:50.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 219466 10.69 MB/sec execute 372 sec latency 44.510 ms 2024-03-24T04:27:51.219 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:51 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:51.219 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:51 smithi022 ceph-mon[66341]: Upgrade: osd.2 is safe to restart 2024-03-24T04:27:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:51 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T04:27:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:51 smithi116 ceph-mon[53251]: Upgrade: osd.2 is safe to restart 2024-03-24T04:27:51.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 219833 10.69 MB/sec execute 373 sec latency 39.882 ms 2024-03-24T04:27:52.241 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:52 smithi022 ceph-mon[66341]: pgmap v124: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 665 op/s 2024-03-24T04:27:52.241 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:52.241 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:27:52.241 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:52 smithi116 ceph-mon[53251]: pgmap v124: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 665 op/s 2024-03-24T04:27:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:27:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T04:27:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:27:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 220198 10.71 MB/sec execute 374 sec latency 44.499 ms 2024-03-24T04:27:53.264 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:53 smithi022 ceph-mon[66341]: Upgrade: Updating osd.2 2024-03-24T04:27:53.265 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:53 smithi022 ceph-mon[66341]: Deploying daemon osd.2 on smithi022 2024-03-24T04:27:53.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:53 smithi116 ceph-mon[53251]: Upgrade: Updating osd.2 2024-03-24T04:27:53.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:53 smithi116 ceph-mon[53251]: Deploying daemon osd.2 on smithi022 2024-03-24T04:27:53.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 220564 10.71 MB/sec execute 375 sec latency 37.805 ms 2024-03-24T04:27:54.295 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:54 smithi022 ceph-mon[66341]: pgmap v125: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-24T04:27:54.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:54 smithi116 ceph-mon[53251]: pgmap v125: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-24T04:27:54.590 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:27:54.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 220858 10.71 MB/sec execute 376 sec latency 48.324 ms 2024-03-24T04:27:55.320 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:55 smithi022 ceph-mon[66341]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:55.320 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:55 smithi022 ceph-mon[66341]: pgmap v126: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 698 op/s 2024-03-24T04:27:55.320 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:55 smithi022 ceph-mon[66341]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:55 smithi116 ceph-mon[53251]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:55 smithi116 ceph-mon[53251]: pgmap v126: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 698 op/s 2024-03-24T04:27:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:55 smithi116 ceph-mon[53251]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:55.609 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:27:55.609 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 28s ago 14m 24.7M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (2m) 28s ago 14m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (2m) 115s ago 14m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (5m) 28s ago 14m 51.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (8m) 28s ago 8m 37.2M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (8m) 115s ago 8m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (4m) 28s ago 16m 501M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (3m) 115s ago 13m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (3m) 28s ago 16m 45.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (3m) 115s ago 13m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (8m) 28s ago 8m 270M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (8m) 115s ago 8m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (5m) 28s ago 14m 19.0M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (4m) 115s ago 13m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:27:55.610 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (86s) 28s ago 13m 248M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (38s) 28s ago 12m 12.7M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (12m) 28s ago 12m 537M 1937M 17.2.0 e1d6a67b021e d07322f3b5f8 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (11m) 28s ago 11m 378M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (11m) 115s ago 11m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (11m) 115s ago 11m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (10m) 115s ago 10m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (10m) 115s ago 10m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:27:55.611 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (4m) 28s ago 13m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:27:55.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 221072 10.70 MB/sec execute 377 sec latency 67.829 ms 2024-03-24T04:27:56.255 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:27:56.256 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-24T04:27:56.257 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T04:27:56.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:27:56.257 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:27:56.360 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:56 smithi022 ceph-mon[66341]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:56 smithi116 ceph-mon[53251]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:27:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "8/23 daemons upgraded", 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:27:56.765 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:27:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: 5 221283 10.68 MB/sec execute 378 sec latency 55.860 ms 2024-03-24T04:27:57.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:57 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/4016237299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:57.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:57 smithi022 ceph-mon[66341]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:57.375 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:57 smithi022 ceph-mon[66341]: pgmap v127: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 527 op/s 2024-03-24T04:27:57.419 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:27:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:57 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/4016237299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:27:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:57 smithi116 ceph-mon[53251]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:27:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:57 smithi116 ceph-mon[53251]: pgmap v127: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 527 op/s 2024-03-24T04:27:57.629 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:57 smithi022 systemd[1]: Stopping Ceph osd.2 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:27:57.853 INFO:teuthology.orchestra.run.smithi022.stdout: 5 221512 10.66 MB/sec execute 379 sec latency 56.529 ms 2024-03-24T04:27:58.138 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:58 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[41885]: 2024-03-24T04:27:58.048+0000 7f050830d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:27:58.139 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:58 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[41885]: 2024-03-24T04:27:58.048+0000 7f050830d700 -1 osd.2 94 *** Got signal Terminated *** 2024-03-24T04:27:58.139 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:58 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[41885]: 2024-03-24T04:27:58.048+0000 7f050830d700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:27:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:58 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3232316202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:27:58.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:58 smithi022 ceph-mon[66341]: osd.2 marked itself down and dead 2024-03-24T04:27:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:58 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3232316202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:27:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:58 smithi116 ceph-mon[53251]: osd.2 marked itself down and dead 2024-03-24T04:27:58.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 221613 10.64 MB/sec execute 380 sec latency 660.338 ms 2024-03-24T04:27:59.197 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:58 smithi022 podman[77276]: 2024-03-24 04:27:58.882127917 +0000 UTC m=+1.055574032 container died d07322f3b5f838c346e7ff441827ef485f9d847c653a76941c1004f987228776 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, com.redhat.component=centos-stream-container, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, version=8, io.openshift.tags=base centos centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T04:27:59.447 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:59 smithi022 podman[77276]: 2024-03-24 04:27:59.437955627 +0000 UTC m=+1.611401743 container cleanup d07322f3b5f838c346e7ff441827ef485f9d847c653a76941c1004f987228776 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T04:27:59.447 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:59 smithi022 bash[77276]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: failed to encode map e95 with expected crc 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: osdmap e95: 8 total, 7 up, 8 in 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: failed to encode map e95 with expected crc 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: pgmap v129: 97 pgs: 9 peering, 6 stale+active+clean, 82 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 510 op/s 2024-03-24T04:27:59.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:27:59 smithi022 ceph-mon[66341]: osdmap e96: 8 total, 7 up, 8 in 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: failed to encode map e95 with expected crc 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: osdmap e95: 8 total, 7 up, 8 in 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: failed to encode map e95 with expected crc 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: pgmap v129: 97 pgs: 9 peering, 6 stale+active+clean, 82 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 510 op/s 2024-03-24T04:27:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:27:59 smithi116 ceph-mon[53251]: osdmap e96: 8 total, 7 up, 8 in 2024-03-24T04:27:59.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:27:59 smithi022 podman[77294]: 2024-03-24 04:27:59.724261984 +0000 UTC m=+0.838249979 container remove d07322f3b5f838c346e7ff441827ef485f9d847c653a76941c1004f987228776 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, distribution-scope=public) 2024-03-24T04:27:59.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 221798 10.62 MB/sec execute 381 sec latency 1078.463 ms 2024-03-24T04:28:00.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:00 smithi022 podman[77379]: 2024-03-24 04:28:00.279567366 +0000 UTC m=+0.027345083 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:00.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:00 smithi022 podman[77379]: 2024-03-24 04:28:00.400484348 +0000 UTC m=+0.148262075 container create 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T04:28:00.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.473 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:00 smithi022 ceph-mon[66341]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:00 smithi116 ceph-mon[53251]: failed to encode map e96 with expected crc 2024-03-24T04:28:00.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:00 smithi022 podman[77379]: 2024-03-24 04:28:00.678504421 +0000 UTC m=+0.426282127 container init 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T04:28:00.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:00 smithi022 podman[77379]: 2024-03-24 04:28:00.687715097 +0000 UTC m=+0.435492804 container start 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T04:28:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 222166 10.61 MB/sec execute 382 sec latency 42.344 ms 2024-03-24T04:28:00.986 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:00 smithi022 podman[77379]: 2024-03-24 04:28:00.82298014 +0000 UTC m=+0.570757853 container attach 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T04:28:01.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:01 smithi022 podman[77379]: 2024-03-24 04:28:01.088599496 +0000 UTC m=+0.836377203 container died 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T04:28:01.490 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:01 smithi022 ceph-mon[66341]: pgmap v131: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 18 peering, 1 stale+active+clean, 73 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 474 op/s; 32/1674 objects degraded (1.912%) 2024-03-24T04:28:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:01 smithi116 ceph-mon[53251]: pgmap v131: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 18 peering, 1 stale+active+clean, 73 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 474 op/s; 32/1674 objects degraded (1.912%) 2024-03-24T04:28:01.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 222436 10.60 MB/sec execute 383 sec latency 82.236 ms 2024-03-24T04:28:02.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:01 smithi022 podman[77403]: 2024-03-24 04:28:01.928470129 +0000 UTC m=+0.834160158 container remove 17bf92a811735a560f1d03bf98678f89e28d853dd71236426a47e95c2f286f26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T04:28:02.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:01 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T04:28:02.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:01 smithi022 systemd[1]: Stopped Ceph osd.2 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:28:02.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:01 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.2.service: Consumed 3min 3.890s CPU time. 2024-03-24T04:28:02.276 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:02 smithi022 ceph-mon[66341]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-24T04:28:02.537 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:02 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 32/1674 objects degraded (1.912%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:02.538 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:02 smithi022 systemd[1]: Starting Ceph osd.2 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:28:02.538 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:02 smithi022 podman[77506]: 2024-03-24 04:28:02.434200355 +0000 UTC m=+0.026335544 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:02.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:02 smithi116 ceph-mon[53251]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-24T04:28:02.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:02 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 32/1674 objects degraded (1.912%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:02.797 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:02 smithi022 podman[77506]: 2024-03-24 04:28:02.578831934 +0000 UTC m=+0.170967106 container create 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T04:28:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 222742 10.59 MB/sec execute 384 sec latency 72.812 ms 2024-03-24T04:28:03.065 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:02 smithi022 podman[77506]: 2024-03-24 04:28:02.953185098 +0000 UTC m=+0.545320279 container init 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T04:28:03.065 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:02 smithi022 podman[77506]: 2024-03-24 04:28:02.958280973 +0000 UTC m=+0.550416148 container start 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:28:03.321 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 podman[77506]: 2024-03-24 04:28:03.133968078 +0000 UTC m=+0.726103257 container attach 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T04:28:03.587 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 bash[77506]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate[77521]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T04:28:03.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:03 smithi022 podman[77506]: 2024-03-24 04:28:03.504499381 +0000 UTC m=+1.096634567 container died 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T04:28:03.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 223029 10.58 MB/sec execute 385 sec latency 86.580 ms 2024-03-24T04:28:04.348 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:04 smithi022 ceph-mon[66341]: pgmap v132: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 18 peering, 1 stale+active+clean, 73 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 MiB/s wr, 150 op/s; 32/1674 objects degraded (1.912%) 2024-03-24T04:28:04.348 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:04.349 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:04 smithi116 ceph-mon[53251]: pgmap v132: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 18 peering, 1 stale+active+clean, 73 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 MiB/s wr, 150 op/s; 32/1674 objects degraded (1.912%) 2024-03-24T04:28:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:04.599 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:04 smithi022 podman[77642]: 2024-03-24 04:28:04.528784029 +0000 UTC m=+1.012017583 container remove 4fe35cf2343b9f7e7192b7041d84a04ee48410a38d10b58431718830cce13e10 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T04:28:04.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 223329 10.57 MB/sec execute 386 sec latency 46.207 ms 2024-03-24T04:28:04.861 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:04 smithi022 podman[77704]: 2024-03-24 04:28:04.67325815 +0000 UTC m=+0.025331706 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:05.124 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:04 smithi022 podman[77704]: 2024-03-24 04:28:04.943756085 +0000 UTC m=+0.295829618 container create 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T04:28:05.378 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:05 smithi022 ceph-mon[66341]: pgmap v133: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.9 MiB/s wr, 462 op/s; 150/1575 objects degraded (9.524%) 2024-03-24T04:28:05.378 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:05 smithi022 ceph-mon[66341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-24T04:28:05.378 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 podman[77704]: 2024-03-24 04:28:05.23952242 +0000 UTC m=+0.591595975 container init 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:28:05.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 podman[77704]: 2024-03-24 04:28:05.245160949 +0000 UTC m=+0.597234483 container start 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True) 2024-03-24T04:28:05.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 ceph-osd[77723]: -- 172.21.15.22:0/64262587 <== mon.0 v2:172.21.15.22:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55643ccc71e0 con 0x55643cc99000 2024-03-24T04:28:05.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 bash[77704]: 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd 2024-03-24T04:28:05.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 systemd[1]: Started Ceph osd.2 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:28:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:05 smithi116 ceph-mon[53251]: pgmap v133: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.9 MiB/s wr, 462 op/s; 150/1575 objects degraded (9.524%) 2024-03-24T04:28:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:05 smithi116 ceph-mon[53251]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-24T04:28:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 223584 10.57 MB/sec execute 387 sec latency 46.110 ms 2024-03-24T04:28:05.887 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:05 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:28:05.811+0000 7fd8078357c0 -1 Falling back to public interface 2024-03-24T04:28:06.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:06.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:06.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 224048 10.60 MB/sec execute 388 sec latency 42.148 ms 2024-03-24T04:28:07.431 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:07 smithi022 ceph-mon[66341]: pgmap v134: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 424 op/s; 147/1566 objects degraded (9.387%) 2024-03-24T04:28:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:07 smithi116 ceph-mon[53251]: pgmap v134: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 424 op/s; 147/1566 objects degraded (9.387%) 2024-03-24T04:28:07.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 224450 10.60 MB/sec execute 389 sec latency 40.712 ms 2024-03-24T04:28:08.437 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:08 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 147/1566 objects degraded (9.387%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:08.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:08 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 147/1566 objects degraded (9.387%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:08.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 224761 10.59 MB/sec execute 390 sec latency 46.138 ms 2024-03-24T04:28:09.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:09 smithi022 ceph-mon[66341]: pgmap v135: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 353 op/s; 150/1668 objects degraded (8.993%) 2024-03-24T04:28:09.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:09 smithi116 ceph-mon[53251]: pgmap v135: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 353 op/s; 150/1668 objects degraded (8.993%) 2024-03-24T04:28:09.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 225112 10.58 MB/sec execute 391 sec latency 39.269 ms 2024-03-24T04:28:10.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 225559 10.58 MB/sec execute 392 sec latency 37.116 ms 2024-03-24T04:28:11.481 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:11 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:28:11.297+0000 7fd8078357c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-24T04:28:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 225887 10.57 MB/sec execute 393 sec latency 56.937 ms 2024-03-24T04:28:12.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:12 smithi022 ceph-mon[66341]: pgmap v136: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 438 op/s; 157/1725 objects degraded (9.101%) 2024-03-24T04:28:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:12 smithi116 ceph-mon[53251]: pgmap v136: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 438 op/s; 157/1725 objects degraded (9.101%) 2024-03-24T04:28:12.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:12 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:28:12.602+0000 7fd8078357c0 -1 osd.2 94 log_to_monitors true 2024-03-24T04:28:12.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 226249 10.56 MB/sec execute 394 sec latency 70.714 ms 2024-03-24T04:28:13.266 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:13 smithi022 ceph-mon[66341]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T04:28:13.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:13 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 157/1725 objects degraded (9.101%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:13.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:13 smithi116 ceph-mon[53251]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T04:28:13.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:13 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 157/1725 objects degraded (9.101%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:13.519 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:28:13 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:28:13.507+0000 7fd7f4c85700 -1 osd.2 94 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:28:13.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 226618 10.56 MB/sec execute 395 sec latency 44.884 ms 2024-03-24T04:28:14.274 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: pgmap v137: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 428 op/s; 157/1725 objects degraded (9.101%) 2024-03-24T04:28:14.274 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T04:28:14.274 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: osdmap e97: 8 total, 7 up, 8 in 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:14 smithi022 ceph-mon[66341]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: pgmap v137: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 428 op/s; 157/1725 objects degraded (9.101%) 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: osdmap e97: 8 total, 7 up, 8 in 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: failed to encode map e97 with expected crc 2024-03-24T04:28:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:14 smithi116 ceph-mon[53251]: from='osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022]' entity='osd.2' 2024-03-24T04:28:14.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 226941 10.56 MB/sec execute 396 sec latency 362.712 ms 2024-03-24T04:28:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:28:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022] boot 2024-03-24T04:28:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:15 smithi116 ceph-mon[53251]: pgmap v140: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.0 MiB/s wr, 526 op/s; 149/1602 objects degraded (9.301%) 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: osd.2 [v2:172.21.15.22:6818/524009022,v1:172.21.15.22:6819/524009022] boot 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:15.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:15 smithi022 ceph-mon[66341]: pgmap v140: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.0 MiB/s wr, 526 op/s; 149/1602 objects degraded (9.301%) 2024-03-24T04:28:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 227079 10.54 MB/sec execute 397 sec latency 1362.757 ms 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:16 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e98 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:16 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: 5 227187 10.53 MB/sec execute 398 sec latency 2129.696 ms 2024-03-24T04:28:17.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.411 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:17 smithi022 ceph-mon[66341]: pgmap v143: 97 pgs: 4 peering, 5 active+undersized, 17 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 766 op/s; 122/1608 objects degraded (7.587%) 2024-03-24T04:28:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e99 with expected crc 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:17.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:17 smithi116 ceph-mon[53251]: pgmap v143: 97 pgs: 4 peering, 5 active+undersized, 17 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.7 MiB/s wr, 766 op/s; 122/1608 objects degraded (7.587%) 2024-03-24T04:28:17.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 227446 10.54 MB/sec execute 399 sec latency 81.206 ms 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: failed to encode map e101 with expected crc 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T04:28:18.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: failed to encode map e101 with expected crc 2024-03-24T04:28:18.182 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 122/1608 objects degraded (7.587%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:18.182 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:18.182 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: failed to encode map e100 with expected crc 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: failed to encode map e101 with expected crc 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: failed to encode map e101 with expected crc 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 122/1608 objects degraded (7.587%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:18.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 227650 10.52 MB/sec execute 400 sec latency 353.131 ms 2024-03-24T04:28:19.198 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: failed to encode map e101 with expected crc 2024-03-24T04:28:19.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T04:28:19.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:19 smithi022 ceph-mon[66341]: pgmap v146: 97 pgs: 7 peering, 5 active+undersized+degraded, 85 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.0 MiB/s wr, 412 op/s; 27/1620 objects degraded (1.667%); 35 KiB/s, 22 objects/s recovering 2024-03-24T04:28:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: failed to encode map e101 with expected crc 2024-03-24T04:28:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T04:28:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: failed to encode map e102 with expected crc 2024-03-24T04:28:19.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:19 smithi116 ceph-mon[53251]: pgmap v146: 97 pgs: 7 peering, 5 active+undersized+degraded, 85 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.0 MiB/s wr, 412 op/s; 27/1620 objects degraded (1.667%); 35 KiB/s, 22 objects/s recovering 2024-03-24T04:28:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 227833 10.52 MB/sec execute 401 sec latency 1011.616 ms 2024-03-24T04:28:20.224 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: failed to encode map e102 with expected crc 2024-03-24T04:28:20.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: failed to encode map e102 with expected crc 2024-03-24T04:28:20.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T04:28:20.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:20 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: failed to encode map e102 with expected crc 2024-03-24T04:28:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: failed to encode map e102 with expected crc 2024-03-24T04:28:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T04:28:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:20 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 228059 10.50 MB/sec execute 402 sec latency 62.616 ms 2024-03-24T04:28:21.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:21 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:21 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:21 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:21 smithi022 ceph-mon[66341]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:21 smithi022 ceph-mon[66341]: pgmap v148: 97 pgs: 3 peering, 94 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.8 MiB/s wr, 258 op/s; 125 KiB/s, 34 objects/s recovering 2024-03-24T04:28:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:21 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:21 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:21 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:21 smithi116 ceph-mon[53251]: failed to encode map e103 with expected crc 2024-03-24T04:28:21.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:21 smithi116 ceph-mon[53251]: pgmap v148: 97 pgs: 3 peering, 94 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.8 MiB/s wr, 258 op/s; 125 KiB/s, 34 objects/s recovering 2024-03-24T04:28:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 228405 10.49 MB/sec execute 403 sec latency 41.326 ms 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1620 objects degraded (1.667%), 5 pgs degraded) 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T04:28:22.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:22 smithi022 ceph-mon[66341]: Upgrade: osd.3 is safe to restart 2024-03-24T04:28:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1620 objects degraded (1.667%), 5 pgs degraded) 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T04:28:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T04:28:22.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:22 smithi116 ceph-mon[53251]: Upgrade: osd.3 is safe to restart 2024-03-24T04:28:22.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 228633 10.48 MB/sec execute 404 sec latency 46.519 ms 2024-03-24T04:28:23.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:23 smithi022 ceph-mon[66341]: pgmap v149: 97 pgs: 3 peering, 94 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.9 MiB/s wr, 198 op/s; 96 KiB/s, 26 objects/s recovering 2024-03-24T04:28:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:23 smithi116 ceph-mon[53251]: pgmap v149: 97 pgs: 3 peering, 94 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.9 MiB/s wr, 198 op/s; 96 KiB/s, 26 objects/s recovering 2024-03-24T04:28:23.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 228872 10.46 MB/sec execute 405 sec latency 47.802 ms 2024-03-24T04:28:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 229183 10.46 MB/sec execute 406 sec latency 103.103 ms 2024-03-24T04:28:25.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 229408 10.44 MB/sec execute 407 sec latency 75.017 ms 2024-03-24T04:28:26.174 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 330 KiB/s, 21 objects/s recovering 2024-03-24T04:28:26.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: Upgrade: Updating osd.3 2024-03-24T04:28:26.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:26.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:28:26.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:26.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:26 smithi022 ceph-mon[66341]: Deploying daemon osd.3 on smithi022 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.7 MiB/s wr, 409 op/s; 330 KiB/s, 21 objects/s recovering 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: Upgrade: Updating osd.3 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:26 smithi116 ceph-mon[53251]: Deploying daemon osd.3 on smithi022 2024-03-24T04:28:26.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 229753 10.42 MB/sec execute 408 sec latency 42.253 ms 2024-03-24T04:28:27.858 INFO:teuthology.orchestra.run.smithi022.stdout: 5 230025 10.42 MB/sec execute 409 sec latency 69.378 ms 2024-03-24T04:28:28.058 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:28:28.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:28 smithi022 ceph-mon[66341]: pgmap v151: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.8 MiB/s wr, 356 op/s; 275 KiB/s, 16 objects/s recovering 2024-03-24T04:28:28.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:28 smithi116 ceph-mon[53251]: pgmap v151: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.8 MiB/s wr, 356 op/s; 275 KiB/s, 16 objects/s recovering 2024-03-24T04:28:28.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 230242 10.41 MB/sec execute 410 sec latency 37.509 ms 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (6m) 14s ago 15m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (2m) 14s ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (2m) 2m ago 14m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (6m) 14s ago 14m 51.7M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (9m) 14s ago 9m 37.3M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:28:29.106 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (9m) 2m ago 9m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (5m) 14s ago 16m 502M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (4m) 14s ago 16m 46.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (3m) 2m ago 14m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (9m) 14s ago 9m 263M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (8m) 2m ago 8m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (6m) 14s ago 14m 18.7M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (5m) 2m ago 14m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (2m) 14s ago 13m 309M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (72s) 14s ago 13m 230M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:28:29.107 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (24s) 14s ago 12m 12.0M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (12m) 14s ago 12m 414M 1937M 17.2.0 e1d6a67b021e 7a2160a18931 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (11m) 2m ago 11m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (11m) 2m ago 11m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (11m) 2m ago 11m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (10m) 2m ago 10m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:28:29.108 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (4m) 14s ago 14m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:28:29.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:29 smithi022 ceph-mon[66341]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:29.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:29 smithi116 ceph-mon[53251]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:28:29.735 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:28:29.736 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:28:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 230709 10.44 MB/sec execute 411 sec latency 37.502 ms 2024-03-24T04:28:30.233 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "9/23 daemons upgraded", 2024-03-24T04:28:30.234 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:28:30.235 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:28:30.235 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:28:30.296 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:30 smithi022 ceph-mon[66341]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:30.296 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:30 smithi022 ceph-mon[66341]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:30.296 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:30 smithi022 ceph-mon[66341]: pgmap v152: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 440 op/s; 237 KiB/s, 6 objects/s recovering 2024-03-24T04:28:30.296 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:30 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3700760701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:30 smithi116 ceph-mon[53251]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:30 smithi116 ceph-mon[53251]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:30 smithi116 ceph-mon[53251]: pgmap v152: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 440 op/s; 237 KiB/s, 6 objects/s recovering 2024-03-24T04:28:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:30 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3700760701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:30.861 INFO:teuthology.orchestra.run.smithi022.stdout: 5 231085 10.44 MB/sec execute 412 sec latency 41.739 ms 2024-03-24T04:28:30.869 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:28:31.081 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:30 smithi022 systemd[1]: Stopping Ceph osd.3 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:28:31.332 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:31 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1207613988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:28:31.332 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:31 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[45736]: 2024-03-24T04:28:31.306+0000 7fcaf8d49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:28:31.332 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:31 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[45736]: 2024-03-24T04:28:31.306+0000 7fcaf8d49700 -1 osd.3 103 *** Got signal Terminated *** 2024-03-24T04:28:31.332 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:31 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[45736]: 2024-03-24T04:28:31.306+0000 7fcaf8d49700 -1 osd.3 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:28:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:31 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1207613988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:28:31.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 231451 10.45 MB/sec execute 413 sec latency 47.377 ms 2024-03-24T04:28:32.336 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:32 smithi022 ceph-mon[66341]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:32.336 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:32 smithi022 ceph-mon[66341]: pgmap v153: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 383 op/s; 204 KiB/s, 3 objects/s recovering 2024-03-24T04:28:32.336 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:32 smithi022 ceph-mon[66341]: osd.3 marked itself down and dead 2024-03-24T04:28:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:32 smithi116 ceph-mon[53251]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:28:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:32 smithi116 ceph-mon[53251]: pgmap v153: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.5 MiB/s wr, 383 op/s; 204 KiB/s, 3 objects/s recovering 2024-03-24T04:28:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:32 smithi116 ceph-mon[53251]: osd.3 marked itself down and dead 2024-03-24T04:28:32.842 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:32 smithi022 podman[79830]: 2024-03-24 04:28:32.833473682 +0000 UTC m=+1.739210845 container died 7a2160a18931e28072202baa286d4f89eff7f6f4ec5d11044f273b8ac57e03a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., ceph=True) 2024-03-24T04:28:32.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 231658 10.43 MB/sec execute 414 sec latency 555.082 ms 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: osdmap e104: 8 total, 7 up, 8 in 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.352 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 3]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 4]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 3]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 2]}]: dispatch 2024-03-24T04:28:33.353 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 3]}]: dispatch 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: osdmap e104: 8 total, 7 up, 8 in 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: failed to encode map e104 with expected crc 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 3]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 4]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 3]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 3]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 2]}]: dispatch 2024-03-24T04:28:33.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 3]}]: dispatch 2024-03-24T04:28:33.629 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:33 smithi022 podman[79830]: 2024-03-24 04:28:33.452874702 +0000 UTC m=+2.358611866 container cleanup 7a2160a18931e28072202baa286d4f89eff7f6f4ec5d11044f273b8ac57e03a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, architecture=x86_64, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-type=git, distribution-scope=public, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , version=8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T04:28:33.629 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:33 smithi022 bash[79830]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3 2024-03-24T04:28:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 231911 10.41 MB/sec execute 415 sec latency 876.307 ms 2024-03-24T04:28:33.882 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:33 smithi022 podman[79852]: 2024-03-24 04:28:33.870112178 +0000 UTC m=+1.033392548 container remove 7a2160a18931e28072202baa286d4f89eff7f6f4ec5d11044f273b8ac57e03a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64) 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: pgmap v155: 97 pgs: 16 stale+active+clean, 81 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 439 op/s; 193 KiB/s, 0 objects/s recovering 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 3]}]': finished 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 3]}]': finished 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 4]}]': finished 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 3]}]': finished 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 3]}]': finished 2024-03-24T04:28:34.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 3]}]': finished 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]': finished 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 2]}]': finished 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 3]}]': finished 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: osdmap e105: 8 total, 7 up, 8 in 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: pgmap v155: 97 pgs: 16 stale+active+clean, 81 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 439 op/s; 193 KiB/s, 0 objects/s recovering 2024-03-24T04:28:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 3]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 3]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 4]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 3]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 3]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 3]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 2]}]': finished 2024-03-24T04:28:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 3]}]': finished 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: osdmap e105: 8 total, 7 up, 8 in 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: failed to encode map e105 with expected crc 2024-03-24T04:28:34.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:34.710 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:34 smithi022 podman[79937]: 2024-03-24 04:28:34.39088018 +0000 UTC m=+0.028757693 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:34.710 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:34 smithi022 podman[79937]: 2024-03-24 04:28:34.540355577 +0000 UTC m=+0.178233082 container create bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T04:28:34.860 INFO:teuthology.orchestra.run.smithi022.stdout: 5 232037 10.39 MB/sec execute 416 sec latency 1527.477 ms 2024-03-24T04:28:35.047 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:34 smithi022 podman[79937]: 2024-03-24 04:28:34.851238097 +0000 UTC m=+0.489115596 container init bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T04:28:35.048 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:34 smithi022 podman[79937]: 2024-03-24 04:28:34.858454567 +0000 UTC m=+0.496332072 container start bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T04:28:35.048 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:34 smithi022 podman[79937]: 2024-03-24 04:28:34.934596412 +0000 UTC m=+0.572473919 container attach bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:28:35.404 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:35 smithi022 conmon[79952]: conmon bf430949cc976553f32b : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f.scope/container/memory.events 2024-03-24T04:28:35.404 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:35 smithi022 podman[79937]: 2024-03-24 04:28:35.254696315 +0000 UTC m=+0.892573822 container died bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:28:35.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: osdmap e106: 8 total, 7 up, 8 in 2024-03-24T04:28:35.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.404 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.405 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:35 smithi022 ceph-mon[66341]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: osdmap e106: 8 total, 7 up, 8 in 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:35 smithi116 ceph-mon[53251]: failed to encode map e106 with expected crc 2024-03-24T04:28:35.861 INFO:teuthology.orchestra.run.smithi022.stdout: 5 232192 10.37 MB/sec execute 417 sec latency 2051.114 ms 2024-03-24T04:28:36.195 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: pgmap v158: 97 pgs: 10 active+undersized, 1 unknown, 7 peering, 19 active+undersized+degraded, 4 stale+active+clean, 56 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 op/s; 166/1647 objects degraded (10.079%) 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 166/1647 objects degraded (10.079%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: osdmap e107: 8 total, 7 up, 8 in 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: 2.1b starting backfill to osd.2 from (0'0,0'0] MAX to 103'7528 2024-03-24T04:28:36.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:36 smithi022 ceph-mon[66341]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.455 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:36 smithi022 podman[79963]: 2024-03-24 04:28:36.415756869 +0000 UTC m=+1.151874399 container remove bf430949cc976553f32b0addcc41ce1a0d4c59a637880fe2a0c7afc54eeb868f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-24T04:28:36.455 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:36 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-24T04:28:36.455 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:36 smithi022 systemd[1]: Stopped Ceph osd.3 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:28:36.455 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:36 smithi022 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.3.service: Consumed 4min 35.761s CPU time. 2024-03-24T04:28:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: pgmap v158: 97 pgs: 10 active+undersized, 1 unknown, 7 peering, 19 active+undersized+degraded, 4 stale+active+clean, 56 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 op/s; 166/1647 objects degraded (10.079%) 2024-03-24T04:28:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 166/1647 objects degraded (10.079%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: osdmap e107: 8 total, 7 up, 8 in 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: 2.1b starting backfill to osd.2 from (0'0,0'0] MAX to 103'7528 2024-03-24T04:28:36.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:36 smithi116 ceph-mon[53251]: failed to encode map e107 with expected crc 2024-03-24T04:28:36.867 INFO:teuthology.orchestra.run.smithi022.stdout: 5 232589 10.39 MB/sec execute 418 sec latency 41.119 ms 2024-03-24T04:28:36.956 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:36 smithi022 systemd[1]: Starting Ceph osd.3 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:28:37.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:37 smithi022 ceph-mon[66341]: failed to encode map e108 with expected crc 2024-03-24T04:28:37.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:37 smithi022 ceph-mon[66341]: osdmap e108: 8 total, 7 up, 8 in 2024-03-24T04:28:37.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:37 smithi022 ceph-mon[66341]: failed to encode map e108 with expected crc 2024-03-24T04:28:37.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:37 smithi022 ceph-mon[66341]: pgmap v161: 97 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 14 active+undersized, 1 unknown, 7 peering, 24 active+undersized+degraded, 49 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 205/1674 objects degraded (12.246%); 608/1674 objects misplaced (36.320%) 2024-03-24T04:28:37.207 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:37 smithi022 podman[80057]: 2024-03-24 04:28:36.983026905 +0000 UTC m=+0.029666727 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:37.457 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:37 smithi022 podman[80057]: 2024-03-24 04:28:37.27585528 +0000 UTC m=+0.322495080 container create f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T04:28:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:37 smithi116 ceph-mon[53251]: failed to encode map e108 with expected crc 2024-03-24T04:28:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:37 smithi116 ceph-mon[53251]: osdmap e108: 8 total, 7 up, 8 in 2024-03-24T04:28:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:37 smithi116 ceph-mon[53251]: failed to encode map e108 with expected crc 2024-03-24T04:28:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:37 smithi116 ceph-mon[53251]: pgmap v161: 97 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 14 active+undersized, 1 unknown, 7 peering, 24 active+undersized+degraded, 49 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 205/1674 objects degraded (12.246%); 608/1674 objects misplaced (36.320%) 2024-03-24T04:28:37.710 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:37 smithi022 podman[80057]: 2024-03-24 04:28:37.688558155 +0000 UTC m=+0.735197968 container init f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T04:28:37.710 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:37 smithi022 podman[80057]: 2024-03-24 04:28:37.697282165 +0000 UTC m=+0.743921970 container start f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T04:28:37.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 233011 10.37 MB/sec execute 419 sec latency 396.055 ms 2024-03-24T04:28:37.961 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:37 smithi022 podman[80057]: 2024-03-24 04:28:37.87500955 +0000 UTC m=+0.921649365 container attach f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T04:28:38.212 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: osdmap e109: 8 total, 7 up, 8 in 2024-03-24T04:28:38.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-mon[66341]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T04:28:38.464 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 bash[80057]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate[80071]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T04:28:38.465 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:38 smithi022 podman[80057]: 2024-03-24 04:28:38.348150455 +0000 UTC m=+1.394790258 container died f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T04:28:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: osdmap e109: 8 total, 7 up, 8 in 2024-03-24T04:28:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:38 smithi116 ceph-mon[53251]: failed to encode map e109 with expected crc 2024-03-24T04:28:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 233370 10.36 MB/sec execute 420 sec latency 723.872 ms 2024-03-24T04:28:39.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: osdmap e110: 8 total, 7 up, 8 in 2024-03-24T04:28:39.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:39 smithi022 ceph-mon[66341]: pgmap v164: 97 pgs: 1 activating+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 3 peering, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 2.37k op/s; 222/1695 objects degraded (13.097%); 644/1695 objects misplaced (37.994%); 0 B/s, 148 objects/s recovering 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: osdmap e110: 8 total, 7 up, 8 in 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: failed to encode map e110 with expected crc 2024-03-24T04:28:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:39 smithi116 ceph-mon[53251]: pgmap v164: 97 pgs: 1 activating+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 3 peering, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 2.37k op/s; 222/1695 objects degraded (13.097%); 644/1695 objects misplaced (37.994%); 0 B/s, 148 objects/s recovering 2024-03-24T04:28:39.483 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:39 smithi022 podman[80193]: 2024-03-24 04:28:39.446986376 +0000 UTC m=+1.092651449 container remove f5d41d0d01adc5bd8a428c3e5a28806b60099e734ee5b4c6a8663864192b3632 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-24T04:28:39.734 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:39 smithi022 podman[80250]: 2024-03-24 04:28:39.624367305 +0000 UTC m=+0.026475733 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:28:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 233679 10.36 MB/sec execute 421 sec latency 34.409 ms 2024-03-24T04:28:39.985 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:39 smithi022 podman[80250]: 2024-03-24 04:28:39.765485063 +0000 UTC m=+0.167593474 container create 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T04:28:40.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-mon[66341]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-mon[66341]: osdmap e111: 8 total, 7 up, 8 in 2024-03-24T04:28:40.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-mon[66341]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-mon[66341]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:40 smithi116 ceph-mon[53251]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:40 smithi116 ceph-mon[53251]: osdmap e111: 8 total, 7 up, 8 in 2024-03-24T04:28:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:40 smithi116 ceph-mon[53251]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:40 smithi116 ceph-mon[53251]: failed to encode map e111 with expected crc 2024-03-24T04:28:40.487 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 podman[80250]: 2024-03-24 04:28:40.243319376 +0000 UTC m=+0.645427787 container init 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:28:40.488 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 podman[80250]: 2024-03-24 04:28:40.252620635 +0000 UTC m=+0.654729057 container start 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:28:40.488 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-osd[80269]: -- 172.21.15.22:0/1250556262 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605bbc351e0 con 0x5605bbc07000 2024-03-24T04:28:40.488 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 bash[80250]: 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a 2024-03-24T04:28:40.488 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 systemd[1]: Started Ceph osd.3 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:28:40.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 234062 10.38 MB/sec execute 422 sec latency 47.085 ms 2024-03-24T04:28:41.027 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:40 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:28:40.844+0000 7f80db0727c0 -1 Falling back to public interface 2024-03-24T04:28:41.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: osdmap e112: 8 total, 7 up, 8 in 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: pgmap v167: 97 pgs: 8 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 13 MiB/s wr, 2.39k op/s; 203/1650 objects degraded (12.303%); 58/1650 objects misplaced (3.515%); 2.0 MiB/s, 2 keys/s, 300 objects/s recovering 2024-03-24T04:28:41.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:41 smithi022 ceph-mon[66341]: osdmap e113: 8 total, 7 up, 8 in 2024-03-24T04:28:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: osdmap e112: 8 total, 7 up, 8 in 2024-03-24T04:28:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: failed to encode map e112 with expected crc 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: pgmap v167: 97 pgs: 8 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 13 MiB/s wr, 2.39k op/s; 203/1650 objects degraded (12.303%); 58/1650 objects misplaced (3.515%); 2.0 MiB/s, 2 keys/s, 300 objects/s recovering 2024-03-24T04:28:41.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:41 smithi116 ceph-mon[53251]: osdmap e113: 8 total, 7 up, 8 in 2024-03-24T04:28:41.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 234410 10.38 MB/sec execute 423 sec latency 56.286 ms 2024-03-24T04:28:42.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:42 smithi022 ceph-mon[66341]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:42 smithi022 ceph-mon[66341]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:42 smithi022 ceph-mon[66341]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:42 smithi022 ceph-mon[66341]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:42 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 203/1650 objects degraded (12.303%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:42 smithi116 ceph-mon[53251]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:42 smithi116 ceph-mon[53251]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:42 smithi116 ceph-mon[53251]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:42 smithi116 ceph-mon[53251]: failed to encode map e113 with expected crc 2024-03-24T04:28:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:42 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 203/1650 objects degraded (12.303%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 234728 10.38 MB/sec execute 424 sec latency 64.303 ms 2024-03-24T04:28:43.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:43 smithi022 ceph-mon[66341]: pgmap v169: 97 pgs: 8 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 1.94k op/s; 203/1650 objects degraded (12.303%); 58/1650 objects misplaced (3.515%); 1.6 MiB/s, 2 keys/s, 244 objects/s recovering 2024-03-24T04:28:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:43 smithi116 ceph-mon[53251]: pgmap v169: 97 pgs: 8 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 24 active+undersized+degraded, 49 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 1.94k op/s; 203/1650 objects degraded (12.303%); 58/1650 objects misplaced (3.515%); 1.6 MiB/s, 2 keys/s, 244 objects/s recovering 2024-03-24T04:28:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 235013 10.37 MB/sec execute 425 sec latency 41.261 ms 2024-03-24T04:28:44.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 235398 10.36 MB/sec execute 426 sec latency 35.568 ms 2024-03-24T04:28:45.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 235805 10.37 MB/sec execute 427 sec latency 32.382 ms 2024-03-24T04:28:46.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:46 smithi022 ceph-mon[66341]: pgmap v170: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 6.0 MiB/s wr, 634 op/s; 195/1692 objects degraded (11.525%); 55/1692 objects misplaced (3.251%); 3.3 MiB/s, 5 keys/s, 105 objects/s recovering 2024-03-24T04:28:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:46 smithi116 ceph-mon[53251]: pgmap v170: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 6.0 MiB/s wr, 634 op/s; 195/1692 objects degraded (11.525%); 55/1692 objects misplaced (3.251%); 3.3 MiB/s, 5 keys/s, 105 objects/s recovering 2024-03-24T04:28:46.547 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:46 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:28:46.474+0000 7f80db0727c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-24T04:28:46.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 236223 10.37 MB/sec execute 428 sec latency 51.157 ms 2024-03-24T04:28:47.305 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:47 smithi022 ceph-mon[66341]: pgmap v171: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.0 MiB/s wr, 343 op/s; 198/1719 objects degraded (11.518%); 59/1719 objects misplaced (3.432%); 1.5 MiB/s, 2 keys/s, 79 objects/s recovering 2024-03-24T04:28:47.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:47 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 198/1719 objects degraded (11.518%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:47.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:47 smithi116 ceph-mon[53251]: pgmap v171: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.0 MiB/s wr, 343 op/s; 198/1719 objects degraded (11.518%); 59/1719 objects misplaced (3.432%); 1.5 MiB/s, 2 keys/s, 79 objects/s recovering 2024-03-24T04:28:47.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:47 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 198/1719 objects degraded (11.518%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:47.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 236622 10.35 MB/sec execute 429 sec latency 50.961 ms 2024-03-24T04:28:48.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-mon[66341]: from='osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:28:48.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-mon[66341]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:28:48.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:48.317 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:48.317 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:28:48.124+0000 7f80db0727c0 -1 osd.3 103 log_to_monitors true 2024-03-24T04:28:48.317 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:28:48 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:28:48.242+0000 7f80cc4ca700 -1 osd.3 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:28:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:48 smithi116 ceph-mon[53251]: from='osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:28:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:48 smithi116 ceph-mon[53251]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T04:28:48.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:48.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:28:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237014 10.35 MB/sec execute 430 sec latency 31.910 ms 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: from='osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:49 smithi022 ceph-mon[66341]: pgmap v173: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 590 op/s; 197/1677 objects degraded (11.747%); 61/1677 objects misplaced (3.637%); 1.5 MiB/s, 2 keys/s, 3 objects/s recovering 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: from='osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: failed to encode map e114 with expected crc 2024-03-24T04:28:49.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:49 smithi116 ceph-mon[53251]: pgmap v173: 97 pgs: 9 active+clean+remapped, 14 active+undersized, 24 active+undersized+degraded, 50 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 590 op/s; 197/1677 objects degraded (11.747%); 61/1677 objects misplaced (3.637%); 1.5 MiB/s, 2 keys/s, 3 objects/s recovering 2024-03-24T04:28:49.868 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237118 10.33 MB/sec execute 431 sec latency 661.512 ms 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: failed to encode map e115 with expected crc 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566] boot 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:28:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:50 smithi022 ceph-mon[66341]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T04:28:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:28:50.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: failed to encode map e115 with expected crc 2024-03-24T04:28:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: osd.3 [v2:172.21.15.22:6826/10115566,v1:172.21.15.22:6827/10115566] boot 2024-03-24T04:28:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T04:28:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T04:28:50.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:50 smithi116 ceph-mon[53251]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T04:28:50.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237118 10.31 MB/sec execute 432 sec latency 1661.546 ms 2024-03-24T04:28:51.491 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.491 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.491 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: 2.1d starting backfill to osd.3 from (0'0,0'0] MAX to 94'10810 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: 2.18 starting backfill to osd.3 from (0'0,0'0] MAX to 113'12861 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: pgmap v176: 97 pgs: 2 remapped+peering, 2 activating, 7 active+clean+remapped, 12 active+undersized, 21 active+undersized+degraded, 53 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.2 MiB/s wr, 485 op/s; 185/1659 objects degraded (11.151%); 45/1659 objects misplaced (2.712%) 2024-03-24T04:28:51.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:51 smithi022 ceph-mon[66341]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T04:28:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: failed to encode map e116 with expected crc 2024-03-24T04:28:51.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: 2.1d starting backfill to osd.3 from (0'0,0'0] MAX to 94'10810 2024-03-24T04:28:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: 2.18 starting backfill to osd.3 from (0'0,0'0] MAX to 113'12861 2024-03-24T04:28:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: pgmap v176: 97 pgs: 2 remapped+peering, 2 activating, 7 active+clean+remapped, 12 active+undersized, 21 active+undersized+degraded, 53 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.2 MiB/s wr, 485 op/s; 185/1659 objects degraded (11.151%); 45/1659 objects misplaced (2.712%) 2024-03-24T04:28:51.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:51 smithi116 ceph-mon[53251]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T04:28:51.869 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237290 10.30 MB/sec execute 433 sec latency 2205.245 ms 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:52.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: failed to encode map e117 with expected crc 2024-03-24T04:28:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:52.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237468 10.30 MB/sec execute 434 sec latency 630.666 ms 2024-03-24T04:28:53.293 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 185/1659 objects degraded (11.151%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:53.294 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:53 smithi022 ceph-mon[66341]: pgmap v179: 97 pgs: 2 remapped+peering, 2 activating, 7 active+clean+remapped, 12 active+undersized, 21 active+undersized+degraded, 53 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s wr, 125 op/s; 185/1659 objects degraded (11.151%); 45/1659 objects misplaced (2.712%) 2024-03-24T04:28:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T04:28:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: failed to encode map e118 with expected crc 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 185/1659 objects degraded (11.151%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:53.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:53 smithi116 ceph-mon[53251]: pgmap v179: 97 pgs: 2 remapped+peering, 2 activating, 7 active+clean+remapped, 12 active+undersized, 21 active+undersized+degraded, 53 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.4 MiB/s wr, 125 op/s; 185/1659 objects degraded (11.151%); 45/1659 objects misplaced (2.712%) 2024-03-24T04:28:53.872 INFO:teuthology.orchestra.run.smithi022.stdout: 5 237699 10.30 MB/sec execute 435 sec latency 1048.606 ms 2024-03-24T04:28:54.325 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.326 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:54 smithi022 ceph-mon[66341]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T04:28:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:54 smithi116 ceph-mon[53251]: failed to encode map e119 with expected crc 2024-03-24T04:28:54.872 INFO:teuthology.orchestra.run.smithi022.stdout: 5 238121 10.30 MB/sec execute 436 sec latency 615.862 ms 2024-03-24T04:28:55.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T04:28:55.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.342 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:55 smithi022 ceph-mon[66341]: pgmap v182: 97 pgs: 2 active+recovery_wait+degraded, 3 peering, 2 remapped+peering, 2 activating, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.0 MiB/s wr, 145 op/s; 652/1581 objects degraded (41.240%); 7/1581 objects misplaced (0.443%); 5.3 MiB/s, 2 keys/s, 54 objects/s recovering 2024-03-24T04:28:55.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T04:28:55.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: failed to encode map e120 with expected crc 2024-03-24T04:28:55.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:55 smithi116 ceph-mon[53251]: pgmap v182: 97 pgs: 2 active+recovery_wait+degraded, 3 peering, 2 remapped+peering, 2 activating, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.0 MiB/s wr, 145 op/s; 652/1581 objects degraded (41.240%); 7/1581 objects misplaced (0.443%); 5.3 MiB/s, 2 keys/s, 54 objects/s recovering 2024-03-24T04:28:55.872 INFO:teuthology.orchestra.run.smithi022.stdout: 5 238408 10.30 MB/sec execute 437 sec latency 564.077 ms 2024-03-24T04:28:56.354 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:56 smithi022 ceph-mon[66341]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.355 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:56 smithi022 ceph-mon[66341]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T04:28:56.355 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:56 smithi022 ceph-mon[66341]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.355 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:56 smithi022 ceph-mon[66341]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.355 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:56 smithi022 ceph-mon[66341]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:56 smithi116 ceph-mon[53251]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:56 smithi116 ceph-mon[53251]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T04:28:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:56 smithi116 ceph-mon[53251]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:56 smithi116 ceph-mon[53251]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:56 smithi116 ceph-mon[53251]: failed to encode map e121 with expected crc 2024-03-24T04:28:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: 5 238648 10.29 MB/sec execute 438 sec latency 1009.802 ms 2024-03-24T04:28:57.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:57 smithi022 ceph-mon[66341]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T04:28:57.376 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:57 smithi022 ceph-mon[66341]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:57 smithi022 ceph-mon[66341]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:57 smithi022 ceph-mon[66341]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:57 smithi022 ceph-mon[66341]: pgmap v185: 97 pgs: 2 active+recovery_wait+degraded, 3 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 362 op/s; 652/1575 objects degraded (41.397%); 7/1575 objects misplaced (0.444%); 5.6 MiB/s, 3 keys/s, 62 objects/s recovering 2024-03-24T04:28:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:57 smithi116 ceph-mon[53251]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T04:28:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:57 smithi116 ceph-mon[53251]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:57 smithi116 ceph-mon[53251]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:57 smithi116 ceph-mon[53251]: failed to encode map e122 with expected crc 2024-03-24T04:28:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:57 smithi116 ceph-mon[53251]: pgmap v185: 97 pgs: 2 active+recovery_wait+degraded, 3 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 362 op/s; 652/1575 objects degraded (41.397%); 7/1575 objects misplaced (0.444%); 5.6 MiB/s, 3 keys/s, 62 objects/s recovering 2024-03-24T04:28:57.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 238925 10.28 MB/sec execute 439 sec latency 38.398 ms 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 652/1575 objects degraded (41.397%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:28:58.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:28:58.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:58 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:28:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 652/1575 objects degraded (41.397%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T04:28:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:28:58.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:58 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-24T04:28:58.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 239247 10.27 MB/sec execute 440 sec latency 51.118 ms 2024-03-24T04:28:59.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:28:59 smithi022 ceph-mon[66341]: pgmap v186: 97 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+remapped+backfill_wait, 92 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 5.5 MiB/s wr, 456 op/s; 16/1686 objects degraded (0.949%); 12/1686 objects misplaced (0.712%); 2.4 MiB/s, 0 keys/s, 111 objects/s recovering 2024-03-24T04:28:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:28:59 smithi116 ceph-mon[53251]: pgmap v186: 97 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+remapped+backfill_wait, 92 active+clean; 346 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 5.5 MiB/s wr, 456 op/s; 16/1686 objects degraded (0.949%); 12/1686 objects misplaced (0.712%); 2.4 MiB/s, 0 keys/s, 111 objects/s recovering 2024-03-24T04:28:59.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 239503 10.26 MB/sec execute 441 sec latency 460.440 ms 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.424 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:00 smithi022 ceph-mon[66341]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:00 smithi116 ceph-mon[53251]: failed to encode map e123 with expected crc 2024-03-24T04:29:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 239825 10.25 MB/sec execute 442 sec latency 930.901 ms 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:01 smithi022 ceph-mon[66341]: pgmap v189: 97 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 394 op/s; 2/1710 objects degraded (0.117%); 9/1710 objects misplaced (0.526%); 2.2 MiB/s, 94 objects/s recovering 2024-03-24T04:29:01.500 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:29:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T04:29:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: failed to encode map e124 with expected crc 2024-03-24T04:29:01.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:01 smithi116 ceph-mon[53251]: pgmap v189: 97 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 394 op/s; 2/1710 objects degraded (0.117%); 9/1710 objects misplaced (0.526%); 2.2 MiB/s, 94 objects/s recovering 2024-03-24T04:29:01.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 240147 10.24 MB/sec execute 443 sec latency 57.035 ms 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:02.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:02 smithi022 ceph-mon[66341]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (7m) 11s ago 15m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (3m) 11s ago 15m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (3m) 3m ago 15m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (6m) 11s ago 15m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (9m) 11s ago 9m 37.4M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (9m) 3m ago 9m 15.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:29:02.453 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (5m) 11s ago 17m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (5m) 3m ago 15m 449M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (4m) 11s ago 17m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (4m) 3m ago 14m 34.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (9m) 11s ago 9m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (9m) 3m ago 9m 87.7M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (6m) 11s ago 15m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (6m) 3m ago 14m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (2m) 11s ago 14m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (105s) 11s ago 13m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (57s) 11s ago 13m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (22s) 11s ago 12m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:29:02.454 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (12m) 3m ago 12m 551M 2704M 17.2.0 e1d6a67b021e fe390ab14656 2024-03-24T04:29:02.455 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (12m) 3m ago 12m 382M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:29:02.455 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (11m) 3m ago 11m 566M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:29:02.455 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (11m) 3m ago 11m 469M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:29:02.455 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (5m) 11s ago 14m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:29:02.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T04:29:02.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: failed to encode map e125 with expected crc 2024-03-24T04:29:02.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:02.625 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:02 smithi116 ceph-mon[53251]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 240440 10.24 MB/sec execute 444 sec latency 52.361 ms 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:29:03.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:29:03.019 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:29:03.454 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:29:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "10/23 daemons upgraded", 2024-03-24T04:29:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:29:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:29:03.456 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:29:03.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:03.470 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded (PG_DEGRADED) 2024-03-24T04:29:03.470 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1674827261' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:03.470 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: pgmap v191: 97 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 383 op/s; 2/1710 objects degraded (0.117%); 9/1710 objects misplaced (0.526%); 2.1 MiB/s, 91 objects/s recovering 2024-03-24T04:29:03.470 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:03.470 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded (PG_DEGRADED) 2024-03-24T04:29:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1674827261' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: pgmap v191: 97 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 383 op/s; 2/1710 objects degraded (0.117%); 9/1710 objects misplaced (0.526%); 2.1 MiB/s, 91 objects/s recovering 2024-03-24T04:29:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:03.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:03.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 240678 10.22 MB/sec execute 445 sec latency 49.758 ms 2024-03-24T04:29:04.023 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded 2024-03-24T04:29:04.024 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded 2024-03-24T04:29:04.024 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1b is active+recovering+undersized+degraded+remapped, acting [0,7] 2024-03-24T04:29:04.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:04 smithi022 ceph-mon[66341]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:04.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:04 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1677364648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:29:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:04 smithi116 ceph-mon[53251]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:04.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:04 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1677364648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:29:04.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 241027 10.24 MB/sec execute 446 sec latency 46.855 ms 2024-03-24T04:29:05.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:05 smithi022 ceph-mon[66341]: pgmap v192: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.1 MiB/s wr, 354 op/s; 3.3 MiB/s, 0 keys/s, 20 objects/s recovering 2024-03-24T04:29:05.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:05 smithi116 ceph-mon[53251]: pgmap v192: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.1 MiB/s wr, 354 op/s; 3.3 MiB/s, 0 keys/s, 20 objects/s recovering 2024-03-24T04:29:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 241284 10.23 MB/sec execute 447 sec latency 88.547 ms 2024-03-24T04:29:06.586 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:06 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded) 2024-03-24T04:29:06.586 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:06 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:29:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:06 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1710 objects degraded (0.117%), 1 pg degraded) 2024-03-24T04:29:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:06 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:29:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 241626 10.24 MB/sec execute 448 sec latency 65.907 ms 2024-03-24T04:29:07.409 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:07 smithi022 ceph-mon[66341]: pgmap v193: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 335 op/s; 1.0 MiB/s, 0 keys/s, 13 objects/s recovering 2024-03-24T04:29:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:07 smithi116 ceph-mon[53251]: pgmap v193: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 335 op/s; 1.0 MiB/s, 0 keys/s, 13 objects/s recovering 2024-03-24T04:29:07.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 241868 10.23 MB/sec execute 449 sec latency 64.250 ms 2024-03-24T04:29:08.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 242189 10.22 MB/sec execute 450 sec latency 41.465 ms 2024-03-24T04:29:09.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 242530 10.22 MB/sec execute 451 sec latency 50.943 ms 2024-03-24T04:29:10.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:10 smithi022 ceph-mon[66341]: pgmap v194: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 282 op/s; 937 KiB/s, 0 keys/s, 12 objects/s recovering 2024-03-24T04:29:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:10 smithi116 ceph-mon[53251]: pgmap v194: 97 pgs: 97 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 282 op/s; 937 KiB/s, 0 keys/s, 12 objects/s recovering 2024-03-24T04:29:10.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 242964 10.22 MB/sec execute 452 sec latency 41.275 ms 2024-03-24T04:29:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: 5 243408 10.22 MB/sec execute 453 sec latency 39.960 ms 2024-03-24T04:29:12.273 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:12 smithi022 ceph-mon[66341]: pgmap v195: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 419 op/s; 824 KiB/s, 0 keys/s, 10 objects/s recovering 2024-03-24T04:29:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:12 smithi116 ceph-mon[53251]: pgmap v195: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 419 op/s; 824 KiB/s, 0 keys/s, 10 objects/s recovering 2024-03-24T04:29:12.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 243821 10.21 MB/sec execute 454 sec latency 37.440 ms 2024-03-24T04:29:13.292 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:13 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:29:13.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:13 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:29:13.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 244142 10.21 MB/sec execute 455 sec latency 39.689 ms 2024-03-24T04:29:14.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:14 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:29:14.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:14 smithi022 ceph-mon[66341]: Upgrade: osd.4 is safe to restart 2024-03-24T04:29:14.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:14 smithi022 ceph-mon[66341]: pgmap v196: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.6 MiB/s wr, 356 op/s; 700 KiB/s, 0 keys/s, 9 objects/s recovering 2024-03-24T04:29:14.306 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:14 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:14 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T04:29:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:14 smithi116 ceph-mon[53251]: Upgrade: osd.4 is safe to restart 2024-03-24T04:29:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:14 smithi116 ceph-mon[53251]: pgmap v196: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.6 MiB/s wr, 356 op/s; 700 KiB/s, 0 keys/s, 9 objects/s recovering 2024-03-24T04:29:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:14 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:14.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 244547 10.22 MB/sec execute 456 sec latency 39.829 ms 2024-03-24T04:29:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 244891 10.22 MB/sec execute 457 sec latency 56.691 ms 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: Upgrade: Updating osd.4 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: Deploying daemon osd.4 on smithi116 2024-03-24T04:29:16.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:15 smithi022 ceph-mon[66341]: pgmap v197: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 550 op/s; 686 KiB/s, 0 keys/s, 9 objects/s recovering 2024-03-24T04:29:16.128 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: Upgrade: Updating osd.4 2024-03-24T04:29:16.129 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:16.129 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T04:29:16.129 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:16.129 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: Deploying daemon osd.4 on smithi116 2024-03-24T04:29:16.129 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:15 smithi116 ceph-mon[53251]: pgmap v197: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 550 op/s; 686 KiB/s, 0 keys/s, 9 objects/s recovering 2024-03-24T04:29:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 245255 10.22 MB/sec execute 458 sec latency 44.480 ms 2024-03-24T04:29:17.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 245457 10.21 MB/sec execute 459 sec latency 70.800 ms 2024-03-24T04:29:18.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:18 smithi022 ceph-mon[66341]: pgmap v198: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 445 op/s 2024-03-24T04:29:18.398 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:18.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:18.426 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-mon[53251]: pgmap v198: 97 pgs: 97 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 445 op/s 2024-03-24T04:29:18.426 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:18.426 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:18.723 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:18 smithi116 systemd[1]: Stopping Ceph osd.4 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:29:18.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 245734 10.21 MB/sec execute 460 sec latency 51.253 ms 2024-03-24T04:29:19.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:19 smithi116 ceph-mon[53251]: osd.4 marked itself down and dead 2024-03-24T04:29:19.225 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[31350]: 2024-03-24T04:29:18.931+0000 7f742d5b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:29:19.225 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[31350]: 2024-03-24T04:29:18.931+0000 7f742d5b7700 -1 osd.4 125 *** Got signal Terminated *** 2024-03-24T04:29:19.225 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[31350]: 2024-03-24T04:29:18.931+0000 7f742d5b7700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:29:19.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:19 smithi022 ceph-mon[66341]: osd.4 marked itself down and dead 2024-03-24T04:29:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 245798 10.19 MB/sec execute 461 sec latency 697.663 ms 2024-03-24T04:29:19.925 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:19 smithi116 podman[57692]: 2024-03-24 04:29:19.64451006 +0000 UTC m=+0.839796030 container died fe390ab14656a55fb5833a1c975c889fd54ea1d5e526d553a61aa4ca13ce18a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD) 2024-03-24T04:29:20.223 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57692]: 2024-03-24 04:29:20.007185343 +0000 UTC m=+1.202471314 container cleanup fe390ab14656a55fb5833a1c975c889fd54ea1d5e526d553a61aa4ca13ce18a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, ceph=True, distribution-scope=public, architecture=x86_64, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-03-24T04:29:20.223 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 bash[57692]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4 2024-03-24T04:29:20.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: pgmap v199: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.1 MiB/s wr, 576 op/s 2024-03-24T04:29:20.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: osdmap e126: 8 total, 7 up, 8 in 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:20 smithi116 ceph-mon[53251]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: pgmap v199: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.1 MiB/s wr, 576 op/s 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: osdmap e126: 8 total, 7 up, 8 in 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.430 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:20 smithi022 ceph-mon[66341]: failed to encode map e126 with expected crc 2024-03-24T04:29:20.652 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57719]: 2024-03-24 04:29:20.283737183 +0000 UTC m=+0.638771455 container remove fe390ab14656a55fb5833a1c975c889fd54ea1d5e526d553a61aa4ca13ce18a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-03-24T04:29:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 245992 10.18 MB/sec execute 462 sec latency 1060.256 ms 2024-03-24T04:29:20.945 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57811]: 2024-03-24 04:29:20.555261998 +0000 UTC m=+0.022375869 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:29:20.945 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57811]: 2024-03-24 04:29:20.684766321 +0000 UTC m=+0.151880184 container create c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:29:21.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T04:29:21.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.200 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:21 smithi022 ceph-mon[66341]: pgmap v202: 97 pgs: 3 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 79 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.4 MiB/s wr, 603 op/s; 26/1683 objects degraded (1.545%) 2024-03-24T04:29:21.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T04:29:21.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: failed to encode map e127 with expected crc 2024-03-24T04:29:21.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:21 smithi116 ceph-mon[53251]: pgmap v202: 97 pgs: 3 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 79 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.4 MiB/s wr, 603 op/s; 26/1683 objects degraded (1.545%) 2024-03-24T04:29:21.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57811]: 2024-03-24 04:29:20.946019912 +0000 UTC m=+0.413133776 container init c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T04:29:21.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:20 smithi116 podman[57811]: 2024-03-24 04:29:20.949419785 +0000 UTC m=+0.416533651 container start c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T04:29:21.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:21 smithi116 podman[57811]: 2024-03-24 04:29:21.054787882 +0000 UTC m=+0.521901753 container attach c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T04:29:21.723 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:21 smithi116 podman[57811]: 2024-03-24 04:29:21.318762382 +0000 UTC m=+0.785876256 container died c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T04:29:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 246240 10.17 MB/sec execute 463 sec latency 47.833 ms 2024-03-24T04:29:22.246 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:22 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 26/1683 objects degraded (1.545%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:22.409 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:22 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 26/1683 objects degraded (1.545%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:22.409 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 podman[57837]: 2024-03-24 04:29:22.151207483 +0000 UTC m=+0.826653383 container remove c6352217003e8892472fe9283d4164aa0cac984b51938b50cfc32b1df34c35d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:22.409 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-24T04:29:22.409 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 systemd[1]: Stopped Ceph osd.4 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:29:22.409 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.4.service: Consumed 4min 26.216s CPU time. 2024-03-24T04:29:22.660 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 systemd[1]: Starting Ceph osd.4 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:29:22.660 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 podman[57933]: 2024-03-24 04:29:22.545118558 +0000 UTC m=+0.020352021 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:29:22.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 246488 10.16 MB/sec execute 464 sec latency 58.249 ms 2024-03-24T04:29:22.930 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 podman[57933]: 2024-03-24 04:29:22.660604645 +0000 UTC m=+0.135838098 container create bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T04:29:23.187 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 podman[57933]: 2024-03-24 04:29:22.930737493 +0000 UTC m=+0.405970943 container init bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T04:29:23.188 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:22 smithi116 podman[57933]: 2024-03-24 04:29:22.935148146 +0000 UTC m=+0.410381599 container start bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T04:29:23.188 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 podman[57933]: 2024-03-24 04:29:23.02272893 +0000 UTC m=+0.497962393 container attach bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:23.273 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:23 smithi022 ceph-mon[66341]: pgmap v203: 97 pgs: 3 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 79 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 303 op/s; 26/1683 objects degraded (1.545%) 2024-03-24T04:29:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-mon[53251]: pgmap v203: 97 pgs: 3 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 79 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 303 op/s; 26/1683 objects degraded (1.545%) 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T04:29:23.827 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 bash[57933]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate[57948]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T04:29:23.828 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:23 smithi116 podman[57933]: 2024-03-24 04:29:23.517008835 +0000 UTC m=+0.992242295 container died bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T04:29:23.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 246809 10.15 MB/sec execute 465 sec latency 51.176 ms 2024-03-24T04:29:24.395 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 podman[58065]: 2024-03-24 04:29:24.135932902 +0000 UTC m=+0.613214663 container remove bce819d38974684a729b23bdd39ca10604a7d69fe8e58a2fb362b5a90497cd00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T04:29:24.395 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 podman[58123]: 2024-03-24 04:29:24.268314821 +0000 UTC m=+0.019424401 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:29:24.648 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 podman[58123]: 2024-03-24 04:29:24.39530636 +0000 UTC m=+0.146415943 container create 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 247198 10.14 MB/sec execute 466 sec latency 44.559 ms 2024-03-24T04:29:24.973 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 podman[58123]: 2024-03-24 04:29:24.648460069 +0000 UTC m=+0.399569650 container init 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T04:29:24.973 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 podman[58123]: 2024-03-24 04:29:24.652104477 +0000 UTC m=+0.403214058 container start 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:24.974 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 ceph-osd[58141]: -- 172.21.15.116:0/4062043686 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5648d3f4f1e0 con 0x5648d3f21000 2024-03-24T04:29:24.974 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 bash[58123]: 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b 2024-03-24T04:29:24.974 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:24 smithi116 systemd[1]: Started Ceph osd.4 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:29:25.473 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:29:25.198+0000 7f25055a47c0 -1 Falling back to public interface 2024-03-24T04:29:25.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 247513 10.14 MB/sec execute 467 sec latency 39.165 ms 2024-03-24T04:29:26.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:25 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:26.018 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:25 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:26.018 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:25 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:29:26.018 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:25 smithi022 ceph-mon[66341]: pgmap v204: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 522 op/s; 219/1617 objects degraded (13.544%) 2024-03-24T04:29:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:25 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:26.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:25 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:25 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:29:26.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:25 smithi116 ceph-mon[53251]: pgmap v204: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 522 op/s; 219/1617 objects degraded (13.544%) 2024-03-24T04:29:26.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 247821 10.13 MB/sec execute 468 sec latency 65.858 ms 2024-03-24T04:29:27.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:27 smithi022 ceph-mon[66341]: pgmap v205: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.4 MiB/s wr, 272 op/s; 220/1626 objects degraded (13.530%) 2024-03-24T04:29:27.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:27 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 220/1626 objects degraded (13.530%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:27 smithi116 ceph-mon[53251]: pgmap v205: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.4 MiB/s wr, 272 op/s; 220/1626 objects degraded (13.530%) 2024-03-24T04:29:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:27 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 220/1626 objects degraded (13.530%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:27.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 248146 10.15 MB/sec execute 469 sec latency 47.378 ms 2024-03-24T04:29:28.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 248515 10.15 MB/sec execute 470 sec latency 76.383 ms 2024-03-24T04:29:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 248812 10.15 MB/sec execute 471 sec latency 53.027 ms 2024-03-24T04:29:30.205 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:30.205 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:30.205 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:30 smithi022 ceph-mon[66341]: pgmap v206: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 423 op/s; 228/1647 objects degraded (13.843%) 2024-03-24T04:29:30.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:30.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:30.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:30 smithi116 ceph-mon[53251]: pgmap v206: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 423 op/s; 228/1647 objects degraded (13.843%) 2024-03-24T04:29:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 248997 10.13 MB/sec execute 472 sec latency 52.992 ms 2024-03-24T04:29:31.501 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:31.502 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:31.502 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:31 smithi022 ceph-mon[66341]: pgmap v207: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 408 op/s; 231/1638 objects degraded (14.103%) 2024-03-24T04:29:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:31 smithi116 ceph-mon[53251]: pgmap v207: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 408 op/s; 231/1638 objects degraded (14.103%) 2024-03-24T04:29:31.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 249264 10.13 MB/sec execute 473 sec latency 47.711 ms 2024-03-24T04:29:32.473 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:29:32.313+0000 7f25055a47c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-24T04:29:32.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 249524 10.12 MB/sec execute 474 sec latency 63.851 ms 2024-03-24T04:29:33.042 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:32 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 231/1638 objects degraded (14.103%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:33.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:32 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 231/1638 objects degraded (14.103%), 26 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:33.617 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:29:33.417+0000 7f25055a47c0 -1 osd.4 125 log_to_monitors true 2024-03-24T04:29:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 249753 10.11 MB/sec execute 475 sec latency 41.786 ms 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: pgmap v208: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.4 MiB/s wr, 367 op/s; 231/1638 objects degraded (14.103%) 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:29:34.316 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.473 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:29:34.175+0000 7f24f69fc700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:29:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: pgmap v208: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.4 MiB/s wr, 367 op/s; 231/1638 objects degraded (14.103%) 2024-03-24T04:29:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:29:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T04:29:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:29:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:34.618 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:29:34.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 249957 10.10 MB/sec execute 476 sec latency 82.096 ms 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: failed to encode map e128 with expected crc 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: failed to encode map e128 with expected crc 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.350 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.351 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.351 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T04:29:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: failed to encode map e128 with expected crc 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: failed to encode map e128 with expected crc 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (7m) 45s ago 16m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (3m) 45s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (3m) 6s ago 15m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (7m) 45s ago 16m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (10m) 45s ago 10m 37.4M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (10m) 6s ago 10m 16.5M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (6m) 45s ago 17m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (5m) 6s ago 15m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (5m) 45s ago 17m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (4m) 6s ago 15m 40.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (10m) 45s ago 10m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:29:35.701 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (10m) 6s ago 10m 88.2M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (7m) 45s ago 15m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (6m) 6s ago 15m 18.7M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (3m) 45s ago 14m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (2m) 45s ago 14m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (91s) 45s ago 13m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (56s) 45s ago 13m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (11s) 6s ago 13m 12.6M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (12m) 6s ago 12m 580M 2704M 17.2.0 e1d6a67b021e 1ca9ea4fcb1f 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (12m) 6s ago 12m 508M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (11m) 6s ago 11m 800M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:29:35.702 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (5m) 45s ago 15m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:29:35.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 250121 10.09 MB/sec execute 477 sec latency 714.246 ms 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:36.309 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 9 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:29:36.310 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:29:36.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:36.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T04:29:36.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:36.381 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: pgmap v210: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 410 op/s; 241/1716 objects degraded (14.044%) 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: failed to encode map e129 with expected crc 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211] boot 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:29:36.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:36 smithi022 ceph-mon[66341]: failed to encode map e129 with expected crc 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: pgmap v210: 97 pgs: 13 active+undersized, 26 active+undersized+degraded, 58 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 410 op/s; 241/1716 objects degraded (14.044%) 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: failed to encode map e129 with expected crc 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: osd.4 [v2:172.21.15.116:6800/2098790211,v1:172.21.15.116:6801/2098790211] boot 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T04:29:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:36 smithi116 ceph-mon[53251]: failed to encode map e129 with expected crc 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "11/23 daemons upgraded", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:29:36.840 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:29:36.841 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:29:36.874 INFO:teuthology.orchestra.run.smithi022.stdout: 5 250349 10.08 MB/sec execute 478 sec latency 1093.841 ms 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.426 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:37 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1618052011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: failed to encode map e130 with expected crc 2024-03-24T04:29:37.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:37 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1618052011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN Degraded data redundancy: 228/1710 objects degraded (13.333%), 24 pgs degraded 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 228/1710 objects degraded (13.333%), 24 pgs degraded 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.5 is active+undersized+degraded, acting [2,0] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-03-24T04:29:37.497 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.18 is active+undersized+degraded, acting [5,3] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-24T04:29:37.498 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-24T04:29:37.499 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-03-24T04:29:37.882 INFO:teuthology.orchestra.run.smithi022.stdout: 5 250533 10.06 MB/sec execute 479 sec latency 1467.543 ms 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: pgmap v213: 97 pgs: 4 peering, 11 active+undersized, 24 active+undersized+degraded, 58 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 267 op/s; 228/1710 objects degraded (13.333%) 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3030258369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:29:38.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 228/1710 objects degraded (13.333%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:38.197 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T04:29:38.197 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: failed to encode map e132 with expected crc 2024-03-24T04:29:38.197 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:38 smithi022 ceph-mon[66341]: failed to encode map e132 with expected crc 2024-03-24T04:29:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:29:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: pgmap v213: 97 pgs: 4 peering, 11 active+undersized, 24 active+undersized+degraded, 58 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 267 op/s; 228/1710 objects degraded (13.333%) 2024-03-24T04:29:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: failed to encode map e131 with expected crc 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3030258369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 228/1710 objects degraded (13.333%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: failed to encode map e132 with expected crc 2024-03-24T04:29:38.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:38 smithi116 ceph-mon[53251]: failed to encode map e132 with expected crc 2024-03-24T04:29:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: 5 250777 10.05 MB/sec execute 480 sec latency 399.025 ms 2024-03-24T04:29:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 251064 10.04 MB/sec execute 481 sec latency 63.943 ms 2024-03-24T04:29:39.999 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T04:29:39.999 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: failed to encode map e133 with expected crc 2024-03-24T04:29:39.999 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.000 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.000 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.000 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:39 smithi022 ceph-mon[66341]: pgmap v217: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 9 active+undersized, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 73 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 2.8 MiB/s wr, 381 op/s; 106/1569 objects degraded (6.756%); 2.4 MiB/s, 29 objects/s recovering 2024-03-24T04:29:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T04:29:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: failed to encode map e133 with expected crc 2024-03-24T04:29:40.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:39 smithi116 ceph-mon[53251]: pgmap v217: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 9 active+undersized, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 73 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 2.8 MiB/s wr, 381 op/s; 106/1569 objects degraded (6.756%); 2.4 MiB/s, 29 objects/s recovering 2024-03-24T04:29:40.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 251277 10.04 MB/sec execute 482 sec latency 142.367 ms 2024-03-24T04:29:41.022 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:40 smithi022 ceph-mon[66341]: failed to encode map e134 with expected crc 2024-03-24T04:29:41.022 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:40 smithi022 ceph-mon[66341]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T04:29:41.022 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:40 smithi022 ceph-mon[66341]: failed to encode map e134 with expected crc 2024-03-24T04:29:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:40 smithi116 ceph-mon[53251]: failed to encode map e134 with expected crc 2024-03-24T04:29:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:40 smithi116 ceph-mon[53251]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T04:29:41.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:40 smithi116 ceph-mon[53251]: failed to encode map e134 with expected crc 2024-03-24T04:29:41.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 251766 10.05 MB/sec execute 483 sec latency 70.932 ms 2024-03-24T04:29:42.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:41 smithi022 ceph-mon[66341]: failed to encode map e135 with expected crc 2024-03-24T04:29:42.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:41 smithi022 ceph-mon[66341]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T04:29:42.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:41 smithi022 ceph-mon[66341]: failed to encode map e135 with expected crc 2024-03-24T04:29:42.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:41 smithi022 ceph-mon[66341]: pgmap v220: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.4 MiB/s wr, 441 op/s; 22/1494 objects degraded (1.473%); 2.5 MiB/s, 60 keys/s, 56 objects/s recovering 2024-03-24T04:29:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:41 smithi116 ceph-mon[53251]: failed to encode map e135 with expected crc 2024-03-24T04:29:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:41 smithi116 ceph-mon[53251]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T04:29:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:41 smithi116 ceph-mon[53251]: failed to encode map e135 with expected crc 2024-03-24T04:29:42.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:41 smithi116 ceph-mon[53251]: pgmap v220: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.4 MiB/s wr, 441 op/s; 22/1494 objects degraded (1.473%); 2.5 MiB/s, 60 keys/s, 56 objects/s recovering 2024-03-24T04:29:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 252118 10.05 MB/sec execute 484 sec latency 50.774 ms 2024-03-24T04:29:43.044 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:42 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 22/1494 objects degraded (1.473%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:43.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:42 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 22/1494 objects degraded (1.473%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T04:29:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 252476 10.06 MB/sec execute 485 sec latency 53.984 ms 2024-03-24T04:29:44.071 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:43 smithi022 ceph-mon[66341]: pgmap v221: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.5 MiB/s wr, 326 op/s; 22/1494 objects degraded (1.473%); 1.9 MiB/s, 44 keys/s, 41 objects/s recovering 2024-03-24T04:29:44.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:43 smithi116 ceph-mon[53251]: pgmap v221: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.5 MiB/s wr, 326 op/s; 22/1494 objects degraded (1.473%); 1.9 MiB/s, 44 keys/s, 41 objects/s recovering 2024-03-24T04:29:44.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 252774 10.05 MB/sec execute 486 sec latency 42.828 ms 2024-03-24T04:29:45.335 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:45 smithi022 ceph-mon[66341]: pgmap v222: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.7 MiB/s wr, 445 op/s; 1.4 MiB/s, 37 keys/s, 34 objects/s recovering 2024-03-24T04:29:45.336 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:45 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1494 objects degraded (1.473%), 3 pgs degraded) 2024-03-24T04:29:45.336 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:45 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:29:45.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:45 smithi116 ceph-mon[53251]: pgmap v222: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.7 MiB/s wr, 445 op/s; 1.4 MiB/s, 37 keys/s, 34 objects/s recovering 2024-03-24T04:29:45.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:45 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1494 objects degraded (1.473%), 3 pgs degraded) 2024-03-24T04:29:45.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:45 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:29:45.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 253084 10.05 MB/sec execute 487 sec latency 42.978 ms 2024-03-24T04:29:46.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 253319 10.04 MB/sec execute 488 sec latency 41.196 ms 2024-03-24T04:29:47.387 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:47 smithi022 ceph-mon[66341]: pgmap v223: 97 pgs: 97 active+clean; 349 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.0 MiB/s wr, 378 op/s; 90 KiB/s, 29 keys/s, 16 objects/s recovering 2024-03-24T04:29:47.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:47 smithi116 ceph-mon[53251]: pgmap v223: 97 pgs: 97 active+clean; 349 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.0 MiB/s wr, 378 op/s; 90 KiB/s, 29 keys/s, 16 objects/s recovering 2024-03-24T04:29:47.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 253566 10.03 MB/sec execute 489 sec latency 57.260 ms 2024-03-24T04:29:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 253885 10.03 MB/sec execute 490 sec latency 48.840 ms 2024-03-24T04:29:49.209 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:49.209 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:49 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:49 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:49 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:29:49.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 254110 10.01 MB/sec execute 491 sec latency 89.379 ms 2024-03-24T04:29:50.214 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:50 smithi116 ceph-mon[53251]: pgmap v224: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 383 op/s; 0 B/s, 25 keys/s, 3 objects/s recovering 2024-03-24T04:29:50.215 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:50 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:50.230 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:50 smithi022 ceph-mon[66341]: pgmap v224: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 383 op/s; 0 B/s, 25 keys/s, 3 objects/s recovering 2024-03-24T04:29:50.230 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:50 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:50.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 254419 10.00 MB/sec execute 492 sec latency 53.660 ms 2024-03-24T04:29:51.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:51 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:51.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:51 smithi022 ceph-mon[66341]: Upgrade: osd.5 is safe to restart 2024-03-24T04:29:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:51 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T04:29:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:51 smithi116 ceph-mon[53251]: Upgrade: osd.5 is safe to restart 2024-03-24T04:29:51.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 254682 10.00 MB/sec execute 493 sec latency 54.691 ms 2024-03-24T04:29:52.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:52 smithi022 ceph-mon[66341]: pgmap v225: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 432 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:52.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:52.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:29:52.289 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:52 smithi116 ceph-mon[53251]: pgmap v225: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 432 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T04:29:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:29:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 254968 9.99 MB/sec execute 494 sec latency 60.751 ms 2024-03-24T04:29:53.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:53 smithi022 ceph-mon[66341]: Upgrade: Updating osd.5 2024-03-24T04:29:53.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:53 smithi022 ceph-mon[66341]: Deploying daemon osd.5 on smithi116 2024-03-24T04:29:53.325 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:53 smithi116 ceph-mon[53251]: Upgrade: Updating osd.5 2024-03-24T04:29:53.325 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:53 smithi116 ceph-mon[53251]: Deploying daemon osd.5 on smithi116 2024-03-24T04:29:53.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 255348 10.00 MB/sec execute 495 sec latency 51.001 ms 2024-03-24T04:29:54.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:54 smithi116 ceph-mon[53251]: pgmap v226: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 370 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:54.306 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:54 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:54.322 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:54 smithi022 ceph-mon[66341]: pgmap v226: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 370 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:54.322 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:54 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:29:54.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 255609 9.99 MB/sec execute 496 sec latency 50.149 ms 2024-03-24T04:29:54.973 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:54 smithi116 systemd[1]: Stopping Ceph osd.5 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:29:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:55 smithi116 ceph-mon[53251]: osd.5 marked itself down and dead 2024-03-24T04:29:55.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:55 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[34397]: 2024-03-24T04:29:55.052+0000 7fe5934a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:29:55.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:55 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[34397]: 2024-03-24T04:29:55.052+0000 7fe5934a9700 -1 osd.5 135 *** Got signal Terminated *** 2024-03-24T04:29:55.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:55 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[34397]: 2024-03-24T04:29:55.052+0000 7fe5934a9700 -1 osd.5 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:29:55.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:55 smithi022 ceph-mon[66341]: osd.5 marked itself down and dead 2024-03-24T04:29:55.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 255658 9.98 MB/sec execute 497 sec latency 734.496 ms 2024-03-24T04:29:56.145 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:55 smithi116 podman[59945]: 2024-03-24 04:29:55.827273501 +0000 UTC m=+0.909360065 container died 1ca9ea4fcb1fedbf60a6854770eda35d3db478f65b9eddd16f0be10bb912fd83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, architecture=x86_64, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , release=754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0) 2024-03-24T04:29:56.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:56 smithi022 ceph-mon[66341]: pgmap v227: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 557 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:56.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:56 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:29:56.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:56 smithi022 ceph-mon[66341]: failed to encode map e136 with expected crc 2024-03-24T04:29:56.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:56 smithi022 ceph-mon[66341]: osdmap e136: 8 total, 7 up, 8 in 2024-03-24T04:29:56.308 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:56 smithi022 ceph-mon[66341]: failed to encode map e136 with expected crc 2024-03-24T04:29:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:56 smithi116 ceph-mon[53251]: pgmap v227: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 557 op/s; 0 B/s, 2 objects/s recovering 2024-03-24T04:29:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:56 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:29:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:56 smithi116 ceph-mon[53251]: failed to encode map e136 with expected crc 2024-03-24T04:29:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:56 smithi116 ceph-mon[53251]: osdmap e136: 8 total, 7 up, 8 in 2024-03-24T04:29:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:56 smithi116 ceph-mon[53251]: failed to encode map e136 with expected crc 2024-03-24T04:29:56.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:56 smithi116 podman[59945]: 2024-03-24 04:29:56.24497428 +0000 UTC m=+1.327060854 container cleanup 1ca9ea4fcb1fedbf60a6854770eda35d3db478f65b9eddd16f0be10bb912fd83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vcs-type=git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-03-24T04:29:56.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:56 smithi116 bash[59945]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5 2024-03-24T04:29:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 255834 9.97 MB/sec execute 498 sec latency 1063.533 ms 2024-03-24T04:29:56.937 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:56 smithi116 podman[59971]: 2024-03-24 04:29:56.496375583 +0000 UTC m=+0.661327375 container remove 1ca9ea4fcb1fedbf60a6854770eda35d3db478f65b9eddd16f0be10bb912fd83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, distribution-scope=public, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, io.buildah.version=1.19.8, vcs-type=git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0) 2024-03-24T04:29:57.216 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:56 smithi116 podman[60061]: 2024-03-24 04:29:56.841493714 +0000 UTC m=+0.019261183 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:29:57.216 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:56 smithi116 podman[60061]: 2024-03-24 04:29:56.955515663 +0000 UTC m=+0.133283118 container create 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:57.216 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:57 smithi116 podman[60061]: 2024-03-24 04:29:57.216721683 +0000 UTC m=+0.394489142 container init 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.217 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:57 smithi116 ceph-mon[53251]: pgmap v230: 97 pgs: 5 peering, 11 stale+active+clean, 81 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.4 MiB/s wr, 408 op/s 2024-03-24T04:29:57.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: failed to encode map e137 with expected crc 2024-03-24T04:29:57.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:57 smithi022 ceph-mon[66341]: pgmap v230: 97 pgs: 5 peering, 11 stale+active+clean, 81 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.4 MiB/s wr, 408 op/s 2024-03-24T04:29:57.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:57 smithi116 podman[60061]: 2024-03-24 04:29:57.219927813 +0000 UTC m=+0.397695273 container start 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T04:29:57.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:57 smithi116 podman[60061]: 2024-03-24 04:29:57.325142404 +0000 UTC m=+0.502909863 container attach 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T04:29:57.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 256136 9.98 MB/sec execute 499 sec latency 86.586 ms 2024-03-24T04:29:57.973 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:57 smithi116 podman[60061]: 2024-03-24 04:29:57.555199559 +0000 UTC m=+0.732967017 container died 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:29:58.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:58 smithi022 ceph-mon[66341]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T04:29:58.296 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:58 smithi116 ceph-mon[53251]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T04:29:58.723 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 podman[60086]: 2024-03-24 04:29:58.2966616 +0000 UTC m=+0.735243759 container remove 38e0ffc109a11f8c6fcbfc1959addb8e8e27ca0a111efed81d44018aa1a9d9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:29:58.723 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-24T04:29:58.724 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 systemd[1]: Stopped Ceph osd.5 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:29:58.724 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.5.service: Consumed 5min 8.764s CPU time. 2024-03-24T04:29:58.724 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 systemd[1]: Starting Ceph osd.5 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:29:58.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 256523 9.98 MB/sec execute 500 sec latency 27.314 ms 2024-03-24T04:29:59.086 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 podman[60187]: 2024-03-24 04:29:58.65347484 +0000 UTC m=+0.018669551 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:29:59.086 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:58 smithi116 podman[60187]: 2024-03-24 04:29:58.814703777 +0000 UTC m=+0.179898481 container create d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:29:59.086 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 podman[60187]: 2024-03-24 04:29:59.085830027 +0000 UTC m=+0.451024746 container init d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:29:59.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:29:59 smithi022 ceph-mon[66341]: pgmap v231: 97 pgs: 5 active+undersized, 17 active+undersized+degraded, 5 peering, 3 stale+active+clean, 67 active+clean; 349 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 448 op/s; 142/1656 objects degraded (8.575%) 2024-03-24T04:29:59.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-mon[53251]: pgmap v231: 97 pgs: 5 active+undersized, 17 active+undersized+degraded, 5 peering, 3 stale+active+clean, 67 active+clean; 349 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 448 op/s; 142/1656 objects degraded (8.575%) 2024-03-24T04:29:59.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 podman[60187]: 2024-03-24 04:29:59.090605187 +0000 UTC m=+0.455799895 container start d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T04:29:59.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 podman[60187]: 2024-03-24 04:29:59.192605111 +0000 UTC m=+0.557799830 container attach d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:29:59.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 256819 9.97 MB/sec execute 501 sec latency 42.895 ms 2024-03-24T04:29:59.905 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T04:29:59.905 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T04:29:59.905 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T04:29:59.905 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 bash[60187]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate[60201]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T04:29:59.906 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:29:59 smithi116 podman[60187]: 2024-03-24 04:29:59.630239693 +0000 UTC m=+0.995434410 container died d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 1 pg peering; Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: [WRN] OSD_DOWN: 1 osds down 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: osd.5 (root=default,host=smithi116) is down 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.2 is stuck peering for 77s, current state peering, last acting [1,6] 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: [WRN] PG_DEGRADED: Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.3 is active+undersized+degraded, acting [2,3] 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.b is active+undersized+degraded, acting [7,4] 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-24T04:30:00.259 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.12 is active+undersized+degraded, acting [3,4] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.d is active+undersized+degraded, acting [7,6] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-24T04:30:00.260 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:00 smithi022 ceph-mon[66341]: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-24T04:30:00.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 podman[60324]: 2024-03-24 04:30:00.340156754 +0000 UTC m=+0.704716539 container remove d53581158ca04d4c2c66c70b759fc27492988a43a92028d7a27baa369e941e26 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T04:30:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 1 pg peering; Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: [WRN] OSD_DOWN: 1 osds down 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: osd.5 (root=default,host=smithi116) is down 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.2 is stuck peering for 77s, current state peering, last acting [1,6] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: [WRN] PG_DEGRADED: Degraded data redundancy: 142/1656 objects degraded (8.575%), 17 pgs degraded 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.3 is active+undersized+degraded, acting [2,3] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.b is active+undersized+degraded, acting [7,4] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.12 is active+undersized+degraded, acting [3,4] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-03-24T04:30:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.d is active+undersized+degraded, acting [7,6] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-24T04:30:00.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-mon[53251]: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-24T04:30:00.878 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 podman[60384]: 2024-03-24 04:30:00.47672214 +0000 UTC m=+0.020887507 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:30:00.878 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 podman[60384]: 2024-03-24 04:30:00.591405368 +0000 UTC m=+0.135570706 container create bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T04:30:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 257298 9.96 MB/sec execute 502 sec latency 33.467 ms 2024-03-24T04:30:01.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 podman[60384]: 2024-03-24 04:30:00.878049841 +0000 UTC m=+0.422215179 container init bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-24T04:30:01.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 podman[60384]: 2024-03-24 04:30:00.882579015 +0000 UTC m=+0.426744360 container start bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T04:30:01.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-osd[60403]: -- 172.21.15.116:0/1991568005 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x5575242fb600 con 0x557524282c00 2024-03-24T04:30:01.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 ceph-osd[60403]: -- 172.21.15.116:0/1991568005 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5575242b11e0 con 0x557524282c00 2024-03-24T04:30:01.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 bash[60384]: bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 2024-03-24T04:30:01.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:00 smithi116 systemd[1]: Started Ceph osd.5 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:30:01.723 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:01 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:30:01.437+0000 7f99aa7ff7c0 -1 Falling back to public interface 2024-03-24T04:30:01.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 257742 9.96 MB/sec execute 503 sec latency 31.045 ms 2024-03-24T04:30:02.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:02 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:02.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:02 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:02.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:02 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:02.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:02 smithi022 ceph-mon[66341]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 5.0 MiB/s wr, 470 op/s; 193/1710 objects degraded (11.287%) 2024-03-24T04:30:02.315 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:02 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:02.316 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:02 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:02.316 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:02 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:02.316 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:02 smithi116 ceph-mon[53251]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 5.0 MiB/s wr, 470 op/s; 193/1710 objects degraded (11.287%) 2024-03-24T04:30:02.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 258099 9.96 MB/sec execute 504 sec latency 47.513 ms 2024-03-24T04:30:03.195 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:03 smithi116 ceph-mon[53251]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T04:30:03.253 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:03 smithi022 ceph-mon[66341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T04:30:03.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 258452 9.96 MB/sec execute 505 sec latency 48.572 ms 2024-03-24T04:30:04.264 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:04 smithi022 ceph-mon[66341]: pgmap v233: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.6 MiB/s wr, 190 op/s; 193/1710 objects degraded (11.287%) 2024-03-24T04:30:04.265 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:04.265 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:04 smithi116 ceph-mon[53251]: pgmap v233: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.6 MiB/s wr, 190 op/s; 193/1710 objects degraded (11.287%) 2024-03-24T04:30:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:04.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 258935 9.97 MB/sec execute 506 sec latency 42.435 ms 2024-03-24T04:30:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 259251 9.97 MB/sec execute 507 sec latency 51.713 ms 2024-03-24T04:30:06.314 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:06.315 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:06 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:06.315 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:06 smithi022 ceph-mon[66341]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 444 op/s; 186/1626 objects degraded (11.439%) 2024-03-24T04:30:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:06 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:06.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:06 smithi116 ceph-mon[53251]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 444 op/s; 186/1626 objects degraded (11.439%) 2024-03-24T04:30:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 259614 9.99 MB/sec execute 508 sec latency 37.327 ms 2024-03-24T04:30:07.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:07 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 186/1626 objects degraded (11.439%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:07.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:07.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:07.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:07 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 186/1626 objects degraded (11.439%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:07.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:07.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:07.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 259931 9.98 MB/sec execute 509 sec latency 75.378 ms 2024-03-24T04:30:08.029 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:30:08.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:08 smithi022 ceph-mon[66341]: pgmap v235: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 407 op/s; 179/1611 objects degraded (11.111%) 2024-03-24T04:30:08.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:08 smithi116 ceph-mon[53251]: pgmap v235: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 407 op/s; 179/1611 objects degraded (11.111%) 2024-03-24T04:30:08.723 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:30:08.455+0000 7f99aa7ff7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-24T04:30:08.888 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260230 9.98 MB/sec execute 510 sec latency 46.985 ms 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (8m) 78s ago 16m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (4m) 78s ago 16m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (4m) 3s ago 16m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (7m) 78s ago 16m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (11m) 78s ago 11m 37.4M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:30:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (10m) 3s ago 10m 16.6M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (7m) 78s ago 18m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (6m) 3s ago 16m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (5m) 78s ago 18m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (5m) 3s ago 16m 42.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (10m) 78s ago 10m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (10m) 3s ago 10m 88.3M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (7m) 78s ago 16m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (7m) 3s ago 15m 21.4M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (3m) 78s ago 15m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (2m) 78s ago 14m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (2m) 78s ago 14m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (89s) 78s ago 14m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (44s) 3s ago 13m 260M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (8s) 3s ago 13m 12.0M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (12m) 3s ago 12m 541M 2704M 17.2.0 e1d6a67b021e c8ca71362987 2024-03-24T04:30:09.048 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (12m) 3s ago 12m 821M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:30:09.049 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (6m) 78s ago 15m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:30:09.128 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:09 smithi022 ceph-mon[66341]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:09 smithi116 ceph-mon[53251]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:30:09.567 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 9 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:30:09.568 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:30:09.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260468 9.97 MB/sec execute 511 sec latency 47.936 ms 2024-03-24T04:30:10.080 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:09 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:30:09.804+0000 7f99aa7ff7c0 -1 osd.5 135 log_to_monitors true 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "12/23 daemons upgraded", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:30:10.121 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:30:10.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:10.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:10.151 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: pgmap v236: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 508 op/s; 176/1671 objects degraded (10.533%) 2024-03-24T04:30:10.152 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3556055036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:10.152 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: from='osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:30:10.152 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:10 smithi022 ceph-mon[66341]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:30:10.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:30:10.106+0000 7f999bc57700 -1 osd.5 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:30:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: pgmap v236: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 508 op/s; 176/1671 objects degraded (10.533%) 2024-03-24T04:30:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3556055036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: from='osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:30:10.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:10 smithi116 ceph-mon[53251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T04:30:10.648 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 176/1671 objects degraded (10.533%), 23 pgs degraded 2024-03-24T04:30:10.648 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T04:30:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: osd.5 (root=default,host=smithi116) is down 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 176/1671 objects degraded (10.533%), 23 pgs degraded 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.3 is active+undersized+degraded, acting [2,3] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.12 is active+undersized+degraded, acting [3,4] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.18 is active+undersized+degraded, acting [4,3] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-24T04:30:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-03-24T04:30:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-24T04:30:10.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260726 9.95 MB/sec execute 512 sec latency 54.225 ms 2024-03-24T04:30:11.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T04:30:11.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: osdmap e138: 8 total, 7 up, 8 in 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: failed to encode map e138 with expected crc 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2203777392' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T04:30:11.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: osdmap e138: 8 total, 7 up, 8 in 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: failed to encode map e138 with expected crc 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2203777392' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:11.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260778 9.94 MB/sec execute 513 sec latency 809.813 ms 2024-03-24T04:30:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-24T04:30:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:30:12.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662] boot 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:12 smithi116 ceph-mon[53251]: pgmap v239: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 351 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 729 op/s; 183/1722 objects degraded (10.627%) 2024-03-24T04:30:12.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:12.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-24T04:30:12.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:30:12.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: osd.5 [v2:172.21.15.116:6808/3248863662,v1:172.21.15.116:6809/3248863662] boot 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: failed to encode map e139 with expected crc 2024-03-24T04:30:12.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:12 smithi022 ceph-mon[66341]: pgmap v239: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 351 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 729 op/s; 183/1722 objects degraded (10.627%) 2024-03-24T04:30:12.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260778 9.92 MB/sec execute 514 sec latency 1809.844 ms 2024-03-24T04:30:13.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 183/1722 objects degraded (10.627%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:13.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T04:30:13.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.368 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:13 smithi116 ceph-mon[53251]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 183/1722 objects degraded (10.627%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:13 smithi022 ceph-mon[66341]: failed to encode map e140 with expected crc 2024-03-24T04:30:13.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 260969 9.91 MB/sec execute 515 sec latency 2325.853 ms 2024-03-24T04:30:14.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T04:30:14.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.175 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.176 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:14 smithi022 ceph-mon[66341]: pgmap v242: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 351 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s wr, 292 op/s; 183/1722 objects degraded (10.627%) 2024-03-24T04:30:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T04:30:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: failed to encode map e141 with expected crc 2024-03-24T04:30:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:14 smithi116 ceph-mon[53251]: pgmap v242: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 351 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s wr, 292 op/s; 183/1722 objects degraded (10.627%) 2024-03-24T04:30:14.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 261219 9.90 MB/sec execute 516 sec latency 91.607 ms 2024-03-24T04:30:15.202 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:15 smithi022 ceph-mon[66341]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T04:30:15.202 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:15 smithi022 ceph-mon[66341]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.202 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:15 smithi022 ceph-mon[66341]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:15 smithi022 ceph-mon[66341]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:15 smithi022 ceph-mon[66341]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:15 smithi116 ceph-mon[53251]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T04:30:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:15 smithi116 ceph-mon[53251]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:15 smithi116 ceph-mon[53251]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:15 smithi116 ceph-mon[53251]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:15 smithi116 ceph-mon[53251]: failed to encode map e142 with expected crc 2024-03-24T04:30:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 261572 9.89 MB/sec execute 517 sec latency 729.547 ms 2024-03-24T04:30:16.235 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.235 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:16 smithi022 ceph-mon[66341]: pgmap v245: 97 pgs: 2 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 4 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 76 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 MiB/s wr, 275 op/s; 144/1527 objects degraded (9.430%); 395 KiB/s, 12 objects/s recovering 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: failed to encode map e143 with expected crc 2024-03-24T04:30:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:16 smithi116 ceph-mon[53251]: pgmap v245: 97 pgs: 2 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 4 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 76 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 MiB/s wr, 275 op/s; 144/1527 objects degraded (9.430%); 395 KiB/s, 12 objects/s recovering 2024-03-24T04:30:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 261855 9.89 MB/sec execute 518 sec latency 994.714 ms 2024-03-24T04:30:17.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T04:30:17.245 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.246 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.246 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.246 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:17 smithi022 ceph-mon[66341]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:17 smithi116 ceph-mon[53251]: failed to encode map e144 with expected crc 2024-03-24T04:30:17.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 262200 9.89 MB/sec execute 519 sec latency 72.120 ms 2024-03-24T04:30:18.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:18 smithi022 ceph-mon[66341]: pgmap v247: 97 pgs: 1 peering, 2 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 91 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.3 MiB/s wr, 323 op/s; 50/1491 objects degraded (3.353%); 1.8 MiB/s, 43 objects/s recovering 2024-03-24T04:30:18.250 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:18 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 144/1527 objects degraded (9.430%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:18 smithi116 ceph-mon[53251]: pgmap v247: 97 pgs: 1 peering, 2 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 91 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.3 MiB/s wr, 323 op/s; 50/1491 objects degraded (3.353%); 1.8 MiB/s, 43 objects/s recovering 2024-03-24T04:30:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:18 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 144/1527 objects degraded (9.430%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:18.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 262529 9.89 MB/sec execute 520 sec latency 40.677 ms 2024-03-24T04:30:19.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:19.277 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 262918 9.90 MB/sec execute 521 sec latency 36.110 ms 2024-03-24T04:30:20.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:20 smithi022 ceph-mon[66341]: pgmap v248: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.4 MiB/s wr, 348 op/s; 1.9 MiB/s, 37 objects/s recovering 2024-03-24T04:30:20.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:20 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/1491 objects degraded (3.353%), 4 pgs degraded) 2024-03-24T04:30:20.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:20 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:30:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:20 smithi116 ceph-mon[53251]: pgmap v248: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.4 MiB/s wr, 348 op/s; 1.9 MiB/s, 37 objects/s recovering 2024-03-24T04:30:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:20 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/1491 objects degraded (3.353%), 4 pgs degraded) 2024-03-24T04:30:20.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:20 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:30:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 263300 9.90 MB/sec execute 522 sec latency 35.680 ms 2024-03-24T04:30:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 263607 9.90 MB/sec execute 523 sec latency 33.974 ms 2024-03-24T04:30:22.325 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:22 smithi022 ceph-mon[66341]: pgmap v249: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 3.2 MiB/s wr, 356 op/s; 2.0 MiB/s, 25 objects/s recovering 2024-03-24T04:30:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:22 smithi116 ceph-mon[53251]: pgmap v249: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 3.2 MiB/s wr, 356 op/s; 2.0 MiB/s, 25 objects/s recovering 2024-03-24T04:30:22.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 263915 9.90 MB/sec execute 524 sec latency 47.850 ms 2024-03-24T04:30:23.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 264159 9.88 MB/sec execute 525 sec latency 61.886 ms 2024-03-24T04:30:24.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:24 smithi022 ceph-mon[66341]: pgmap v250: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 2.7 MiB/s wr, 304 op/s; 1.7 MiB/s, 22 objects/s recovering 2024-03-24T04:30:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:24 smithi116 ceph-mon[53251]: pgmap v250: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 2.7 MiB/s wr, 304 op/s; 1.7 MiB/s, 22 objects/s recovering 2024-03-24T04:30:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 264415 9.87 MB/sec execute 526 sec latency 60.454 ms 2024-03-24T04:30:25.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 264712 9.87 MB/sec execute 527 sec latency 54.041 ms 2024-03-24T04:30:26.196 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:26 smithi022 ceph-mon[66341]: pgmap v251: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 403 op/s; 1.4 MiB/s, 17 objects/s recovering 2024-03-24T04:30:26.197 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:26 smithi116 ceph-mon[53251]: pgmap v251: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 403 op/s; 1.4 MiB/s, 17 objects/s recovering 2024-03-24T04:30:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:26.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 265179 9.87 MB/sec execute 528 sec latency 33.584 ms 2024-03-24T04:30:27.224 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:27 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:27.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:27 smithi022 ceph-mon[66341]: Upgrade: osd.6 is safe to restart 2024-03-24T04:30:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:27 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T04:30:27.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:27 smithi116 ceph-mon[53251]: Upgrade: osd.6 is safe to restart 2024-03-24T04:30:27.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 265544 9.87 MB/sec execute 529 sec latency 47.034 ms 2024-03-24T04:30:28.235 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:28 smithi022 ceph-mon[66341]: pgmap v252: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.4 MiB/s wr, 445 op/s; 769 KiB/s, 4 objects/s recovering 2024-03-24T04:30:28.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:28 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:28.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:28 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:30:28.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:28 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:28.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:28 smithi116 ceph-mon[53251]: pgmap v252: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.4 MiB/s wr, 445 op/s; 769 KiB/s, 4 objects/s recovering 2024-03-24T04:30:28.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:28 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:28.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:28 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T04:30:28.416 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:28 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:28.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 265919 9.88 MB/sec execute 530 sec latency 40.109 ms 2024-03-24T04:30:29.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:29 smithi022 ceph-mon[66341]: Upgrade: Updating osd.6 2024-03-24T04:30:29.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:29 smithi022 ceph-mon[66341]: Deploying daemon osd.6 on smithi116 2024-03-24T04:30:29.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:29 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:29.394 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:29 smithi116 ceph-mon[53251]: Upgrade: Updating osd.6 2024-03-24T04:30:29.394 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:29 smithi116 ceph-mon[53251]: Deploying daemon osd.6 on smithi116 2024-03-24T04:30:29.394 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:29 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: 5 266290 9.87 MB/sec execute 531 sec latency 45.166 ms 2024-03-24T04:30:30.279 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:30 smithi022 ceph-mon[66341]: pgmap v253: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.6 MiB/s wr, 546 op/s; 706 KiB/s, 4 objects/s recovering 2024-03-24T04:30:30.464 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:30 smithi116 ceph-mon[53251]: pgmap v253: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.6 MiB/s wr, 546 op/s; 706 KiB/s, 4 objects/s recovering 2024-03-24T04:30:30.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 266605 9.88 MB/sec execute 532 sec latency 47.762 ms 2024-03-24T04:30:31.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:31 smithi116 systemd[1]: Stopping Ceph osd.6 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:30:31.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 266856 9.88 MB/sec execute 533 sec latency 55.378 ms 2024-03-24T04:30:32.179 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[37393]: 2024-03-24T04:30:31.851+0000 7f461de4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:30:32.180 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[37393]: 2024-03-24T04:30:31.851+0000 7f461de4d700 -1 osd.6 144 *** Got signal Terminated *** 2024-03-24T04:30:32.180 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[37393]: 2024-03-24T04:30:31.851+0000 7f461de4d700 -1 osd.6 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:30:32.324 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:32 smithi022 ceph-mon[66341]: pgmap v254: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.2 MiB/s wr, 548 op/s; 341 KiB/s, 0 objects/s recovering 2024-03-24T04:30:32.324 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:32 smithi022 ceph-mon[66341]: osd.6 marked itself down and dead 2024-03-24T04:30:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:32 smithi116 ceph-mon[53251]: pgmap v254: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.2 MiB/s wr, 548 op/s; 341 KiB/s, 0 objects/s recovering 2024-03-24T04:30:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:32 smithi116 ceph-mon[53251]: osd.6 marked itself down and dead 2024-03-24T04:30:32.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 267052 9.87 MB/sec execute 534 sec latency 718.017 ms 2024-03-24T04:30:33.102 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:32 smithi116 podman[62211]: 2024-03-24 04:30:32.828083503 +0000 UTC m=+1.329410292 container died c8ca71362987f53b6d1ebe2afcd2aab15385f8ecd72a87cd16bf17402a5f187b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream) 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: osdmap e145: 8 total, 7 up, 8 in 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: pgmap v256: 97 pgs: 10 stale+active+clean, 87 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:33.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:33.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:33 smithi116 podman[62211]: 2024-03-24 04:30:33.293763826 +0000 UTC m=+1.795090622 container cleanup c8ca71362987f53b6d1ebe2afcd2aab15385f8ecd72a87cd16bf17402a5f187b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True, vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container) 2024-03-24T04:30:33.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:33 smithi116 bash[62211]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: osdmap e145: 8 total, 7 up, 8 in 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: failed to encode map e145 with expected crc 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: pgmap v256: 97 pgs: 10 stale+active+clean, 87 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 267206 9.86 MB/sec execute 535 sec latency 1033.904 ms 2024-03-24T04:30:33.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:33 smithi116 podman[62228]: 2024-03-24 04:30:33.662315924 +0000 UTC m=+0.835206883 container remove c8ca71362987f53b6d1ebe2afcd2aab15385f8ecd72a87cd16bf17402a5f187b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-03-24T04:30:34.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: osdmap e146: 8 total, 7 up, 8 in 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [0, 6]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 6]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 6]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 6]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 6]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 1]}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.3"}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]: dispatch 2024-03-24T04:30:34.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1d"}]: dispatch 2024-03-24T04:30:34.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: osdmap e146: 8 total, 7 up, 8 in 2024-03-24T04:30:34.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.366 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: failed to encode map e146 with expected crc 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [0, 6]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 6]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 6]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 6]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 6]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 1]}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.3"}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1d"}]: dispatch 2024-03-24T04:30:34.367 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.015976933 +0000 UTC m=+0.021539966 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:30:34.368 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.138360955 +0000 UTC m=+0.143923961 container create 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T04:30:34.368 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.366406668 +0000 UTC m=+0.371969689 container init 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T04:30:34.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.370313888 +0000 UTC m=+0.375876901 container start 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True) 2024-03-24T04:30:34.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.47489747 +0000 UTC m=+0.480460488 container attach 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T04:30:34.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 267458 9.85 MB/sec execute 536 sec latency 497.095 ms 2024-03-24T04:30:35.141 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:34 smithi116 podman[62314]: 2024-03-24 04:30:34.725636192 +0000 UTC m=+0.731199201 container died 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [0, 6]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 6]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 6]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 6]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 6]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 1]}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.3"}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]': finished 2024-03-24T04:30:35.221 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1d"}]': finished 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: osdmap e147: 8 total, 7 up, 8 in 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:35 smithi022 ceph-mon[66341]: pgmap v259: 97 pgs: 7 active+undersized, 17 active+undersized+degraded, 3 stale+active+clean, 70 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 295 op/s; 138/1650 objects degraded (8.364%) 2024-03-24T04:30:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [0, 6]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 6]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 6]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 6]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 6]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 1]}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.3"}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]': finished 2024-03-24T04:30:35.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1d"}]': finished 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: osdmap e147: 8 total, 7 up, 8 in 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: failed to encode map e147 with expected crc 2024-03-24T04:30:35.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:35 smithi116 ceph-mon[53251]: pgmap v259: 97 pgs: 7 active+undersized, 17 active+undersized+degraded, 3 stale+active+clean, 70 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 295 op/s; 138/1650 objects degraded (8.364%) 2024-03-24T04:30:35.799 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:35 smithi116 podman[62340]: 2024-03-24 04:30:35.492480179 +0000 UTC m=+0.760921367 container remove 36cbec143ae25e88d3e08268684d668565e86ee6dec71cea85704253c444814a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T04:30:35.799 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:35 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-24T04:30:35.799 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:35 smithi116 systemd[1]: Stopped Ceph osd.6 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:30:35.799 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:35 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.6.service: Consumed 4min 49.481s CPU time. 2024-03-24T04:30:35.891 INFO:teuthology.orchestra.run.smithi022.stdout: 5 267633 9.84 MB/sec execute 537 sec latency 1497.146 ms 2024-03-24T04:30:36.183 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:35 smithi116 systemd[1]: Starting Ceph osd.6 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:30:36.183 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:35.930131717 +0000 UTC m=+0.018805607 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:30:36.183 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:36.04887836 +0000 UTC m=+0.137552225 container create 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:30:36.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 138/1650 objects degraded (8.364%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:36.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.262 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: osdmap e148: 8 total, 7 up, 8 in 2024-03-24T04:30:36.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:36 smithi022 ceph-mon[66341]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 138/1650 objects degraded (8.364%), 17 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: osdmap e148: 8 total, 7 up, 8 in 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-mon[53251]: failed to encode map e148 with expected crc 2024-03-24T04:30:36.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:36.28541672 +0000 UTC m=+0.374090609 container init 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T04:30:36.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:36.28938821 +0000 UTC m=+0.378062075 container start 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:30:36.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:36.393811018 +0000 UTC m=+0.482484885 container attach 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T04:30:36.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 267819 9.84 MB/sec execute 538 sec latency 1845.172 ms 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T04:30:37.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 bash[62436]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate[62452]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T04:30:37.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:36 smithi116 podman[62436]: 2024-03-24 04:30:36.886341063 +0000 UTC m=+0.975014930 container died 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T04:30:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:37 smithi022 ceph-mon[66341]: failed to encode map e149 with expected crc 2024-03-24T04:30:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:37 smithi022 ceph-mon[66341]: osdmap e149: 8 total, 7 up, 8 in 2024-03-24T04:30:37.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:37 smithi022 ceph-mon[66341]: failed to encode map e149 with expected crc 2024-03-24T04:30:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:37 smithi022 ceph-mon[66341]: 2.1d starting backfill to osd.7 from (0'0,0'0] MAX to 144'12035 2024-03-24T04:30:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:37 smithi022 ceph-mon[66341]: pgmap v262: 97 pgs: 3 remapped+peering, 1 peering, 10 active+undersized, 25 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 447 op/s; 198/1626 objects degraded (12.177%) 2024-03-24T04:30:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:37 smithi116 ceph-mon[53251]: failed to encode map e149 with expected crc 2024-03-24T04:30:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:37 smithi116 ceph-mon[53251]: osdmap e149: 8 total, 7 up, 8 in 2024-03-24T04:30:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:37 smithi116 ceph-mon[53251]: failed to encode map e149 with expected crc 2024-03-24T04:30:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:37 smithi116 ceph-mon[53251]: 2.1d starting backfill to osd.7 from (0'0,0'0] MAX to 144'12035 2024-03-24T04:30:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:37 smithi116 ceph-mon[53251]: pgmap v262: 97 pgs: 3 remapped+peering, 1 peering, 10 active+undersized, 25 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 447 op/s; 198/1626 objects degraded (12.177%) 2024-03-24T04:30:37.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 268142 9.82 MB/sec execute 539 sec latency 110.414 ms 2024-03-24T04:30:37.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:37 smithi116 podman[62584]: 2024-03-24 04:30:37.6491187 +0000 UTC m=+0.753956404 container remove 299946f33c09ce86c7dbf232385a2c9b6deeec5dc0aeb0f4249e7624e592c7ce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:30:37.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:37 smithi116 podman[62644]: 2024-03-24 04:30:37.772285479 +0000 UTC m=+0.020652562 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:30:37.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:37 smithi116 podman[62644]: 2024-03-24 04:30:37.892022572 +0000 UTC m=+0.140389677 container create 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T04:30:38.307 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:38 smithi022 ceph-mon[66341]: failed to encode map e150 with expected crc 2024-03-24T04:30:38.307 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:38 smithi022 ceph-mon[66341]: osdmap e150: 8 total, 7 up, 8 in 2024-03-24T04:30:38.307 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:38 smithi022 ceph-mon[66341]: failed to encode map e150 with expected crc 2024-03-24T04:30:38.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:38 smithi116 ceph-mon[53251]: failed to encode map e150 with expected crc 2024-03-24T04:30:38.386 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:38 smithi116 ceph-mon[53251]: osdmap e150: 8 total, 7 up, 8 in 2024-03-24T04:30:38.387 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:38 smithi116 ceph-mon[53251]: failed to encode map e150 with expected crc 2024-03-24T04:30:38.387 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 podman[62644]: 2024-03-24 04:30:38.111682224 +0000 UTC m=+0.360049279 container init 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T04:30:38.387 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 podman[62644]: 2024-03-24 04:30:38.115908199 +0000 UTC m=+0.364275261 container start 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b) 2024-03-24T04:30:38.387 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 ceph-osd[62663]: -- 172.21.15.116:0/1177019623 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcad7f31e0 con 0x55bcad7c4c00 2024-03-24T04:30:38.715 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 bash[62644]: 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 2024-03-24T04:30:38.715 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 systemd[1]: Started Ceph osd.6 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:30:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 268487 9.81 MB/sec execute 540 sec latency 617.759 ms 2024-03-24T04:30:38.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:38 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:30:38.715+0000 7f4107ae67c0 -1 Falling back to public interface 2024-03-24T04:30:39.320 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: failed to encode map e151 with expected crc 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: osdmap e151: 8 total, 7 up, 8 in 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: failed to encode map e151 with expected crc 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:39.321 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:39 smithi022 ceph-mon[66341]: pgmap v265: 97 pgs: 3 active+clean+remapped, 4 remapped+peering, 1 peering, 10 active+undersized, 23 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.8 MiB/s wr, 543 op/s; 173/1623 objects degraded (10.659%); 22/1623 objects misplaced (1.356%); 2.0 MiB/s, 3 keys/s, 180 objects/s recovering 2024-03-24T04:30:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: failed to encode map e151 with expected crc 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: osdmap e151: 8 total, 7 up, 8 in 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: failed to encode map e151 with expected crc 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:39 smithi116 ceph-mon[53251]: pgmap v265: 97 pgs: 3 active+clean+remapped, 4 remapped+peering, 1 peering, 10 active+undersized, 23 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.8 MiB/s wr, 543 op/s; 173/1623 objects degraded (10.659%); 22/1623 objects misplaced (1.356%); 2.0 MiB/s, 3 keys/s, 180 objects/s recovering 2024-03-24T04:30:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 268809 9.82 MB/sec execute 541 sec latency 960.869 ms 2024-03-24T04:30:40.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:40 smithi022 ceph-mon[66341]: failed to encode map e152 with expected crc 2024-03-24T04:30:40.346 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:40 smithi022 ceph-mon[66341]: osdmap e152: 8 total, 7 up, 8 in 2024-03-24T04:30:40.347 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:40 smithi022 ceph-mon[66341]: failed to encode map e152 with expected crc 2024-03-24T04:30:40.462 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:40 smithi116 ceph-mon[53251]: failed to encode map e152 with expected crc 2024-03-24T04:30:40.462 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:40 smithi116 ceph-mon[53251]: osdmap e152: 8 total, 7 up, 8 in 2024-03-24T04:30:40.462 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:40 smithi116 ceph-mon[53251]: failed to encode map e152 with expected crc 2024-03-24T04:30:40.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 269122 9.81 MB/sec execute 542 sec latency 58.502 ms 2024-03-24T04:30:41.273 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:41 smithi022 ceph-mon[66341]: pgmap v267: 97 pgs: 7 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.7 MiB/s wr, 343 op/s; 159/1641 objects degraded (9.689%); 50/1641 objects misplaced (3.047%); 6.5 MiB/s, 3 keys/s, 149 objects/s recovering 2024-03-24T04:30:41.283 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:30:41.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:41 smithi116 ceph-mon[53251]: pgmap v267: 97 pgs: 7 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.7 MiB/s wr, 343 op/s; 159/1641 objects degraded (9.689%); 50/1641 objects misplaced (3.047%); 6.5 MiB/s, 3 keys/s, 149 objects/s recovering 2024-03-24T04:30:41.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 269519 9.82 MB/sec execute 543 sec latency 55.465 ms 2024-03-24T04:30:42.311 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:42 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 159/1641 objects degraded (9.689%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:42.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:42 smithi022 ceph-mon[66341]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:42.312 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:42 smithi022 ceph-mon[66341]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (8m) 111s ago 17m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (4m) 111s ago 17m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (4m) 37s ago 16m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (8m) 111s ago 17m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (11m) 111s ago 11m 37.4M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (11m) 37s ago 11m 16.6M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (7m) 111s ago 18m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (6m) 37s ago 16m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:30:42.366 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (6m) 111s ago 18m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (5m) 37s ago 16m 42.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (11m) 111s ago 11m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (11m) 37s ago 11m 88.3M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (8m) 111s ago 17m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (7m) 37s ago 16m 21.4M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (4m) 111s ago 15m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (3m) 111s ago 15m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (2m) 111s ago 15m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (2m) 111s ago 14m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (78s) 37s ago 14m 260M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (41s) 37s ago 13m 12.0M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:30:42.367 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 starting - - - 2704M 2024-03-24T04:30:42.368 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (13m) 37s ago 13m 821M 2704M 17.2.0 e1d6a67b021e da6d025024ec 2024-03-24T04:30:42.368 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (6m) 111s ago 16m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:30:42.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:42 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 159/1641 objects degraded (9.689%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:42.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:42 smithi116 ceph-mon[53251]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:42.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:42 smithi116 ceph-mon[53251]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 269885 9.82 MB/sec execute 544 sec latency 38.722 ms 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:30:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 10 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:30:43.016 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:30:43.570 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:30:43.570 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:30:43.570 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:30:43.570 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "13/23 daemons upgraded", 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:30:43.571 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:30:43.577 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:43 smithi022 ceph-mon[66341]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:43.578 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:43.578 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:43.578 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:43 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/475307058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:43.578 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:43 smithi022 ceph-mon[66341]: pgmap v268: 97 pgs: 7 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 282 op/s; 159/1641 objects degraded (9.689%); 50/1641 objects misplaced (3.047%); 5.3 MiB/s, 2 keys/s, 122 objects/s recovering 2024-03-24T04:30:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:43 smithi116 ceph-mon[53251]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:43 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/475307058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:43.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:43 smithi116 ceph-mon[53251]: pgmap v268: 97 pgs: 7 active+clean+remapped, 1 remapped+peering, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 282 op/s; 159/1641 objects degraded (9.689%); 50/1641 objects misplaced (3.047%); 5.3 MiB/s, 2 keys/s, 122 objects/s recovering 2024-03-24T04:30:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 270165 9.83 MB/sec execute 545 sec latency 47.421 ms 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 159/1641 objects degraded (9.689%), 22 pgs degraded 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: osd.6 (root=default,host=smithi116) is down 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 159/1641 objects degraded (9.689%), 22 pgs degraded 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-03-24T04:30:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1b is active+undersized+degraded, acting [3,2] 2024-03-24T04:30:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-03-24T04:30:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-03-24T04:30:44.589 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:44 smithi022 ceph-mon[66341]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:44.590 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:44.590 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:44.590 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:44 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/174388215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:30:44.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 270523 9.83 MB/sec execute 546 sec latency 49.488 ms 2024-03-24T04:30:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:44 smithi116 ceph-mon[53251]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:30:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:44 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:44.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:44 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:44.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:44 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/174388215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:30:45.269 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:44 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:30:44.995+0000 7f4107ae67c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-24T04:30:45.608 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:45 smithi022 ceph-mon[66341]: pgmap v269: 97 pgs: 8 active+clean+remapped, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 531 op/s; 172/1674 objects degraded (10.275%); 66/1674 objects misplaced (3.943%); 4.0 MiB/s, 2 keys/s, 92 objects/s recovering 2024-03-24T04:30:45.862 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:45 smithi116 ceph-mon[53251]: pgmap v269: 97 pgs: 8 active+clean+remapped, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 531 op/s; 172/1674 objects degraded (10.275%); 66/1674 objects misplaced (3.943%); 4.0 MiB/s, 2 keys/s, 92 objects/s recovering 2024-03-24T04:30:45.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 270831 9.82 MB/sec execute 547 sec latency 46.341 ms 2024-03-24T04:30:46.576 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:46 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:30:46.284+0000 7f4107ae67c0 -1 osd.6 144 log_to_monitors true 2024-03-24T04:30:46.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:46 smithi022 ceph-mon[66341]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:30:46.882 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:46 smithi116 ceph-mon[53251]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T04:30:46.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 271090 9.82 MB/sec execute 548 sec latency 217.388 ms 2024-03-24T04:30:47.166 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:30:47.080+0000 7f40f4f36700 -1 osd.6 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:30:47.715 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: failed to encode map e153 with expected crc 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: osdmap e153: 8 total, 7 up, 8 in 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: failed to encode map e153 with expected crc 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: pgmap v271: 97 pgs: 8 active+clean+remapped, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.1 MiB/s wr, 369 op/s; 176/1674 objects degraded (10.514%); 67/1674 objects misplaced (4.002%); 3.0 MiB/s, 0 keys/s, 1 objects/s recovering 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:47.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:47 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:30:47.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 271490 9.81 MB/sec execute 549 sec latency 255.161 ms 2024-03-24T04:30:47.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T04:30:47.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: failed to encode map e153 with expected crc 2024-03-24T04:30:47.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: osdmap e153: 8 total, 7 up, 8 in 2024-03-24T04:30:47.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: failed to encode map e153 with expected crc 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943]' entity='osd.6' 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: pgmap v271: 97 pgs: 8 active+clean+remapped, 10 active+undersized, 22 active+undersized+degraded, 57 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.1 MiB/s wr, 369 op/s; 176/1674 objects degraded (10.514%); 67/1674 objects misplaced (4.002%); 3.0 MiB/s, 0 keys/s, 1 objects/s recovering 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:30:47.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:30:47.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:47 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:30:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 271679 9.80 MB/sec execute 550 sec latency 801.320 ms 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 172/1674 objects degraded (10.275%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943] boot 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:30:48.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:48.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 172/1674 objects degraded (10.275%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: osd.6 [v2:172.21.15.116:6816/4153478943,v1:172.21.15.116:6817/4153478943] boot 2024-03-24T04:30:49.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T04:30:49.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T04:30:49.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:30:49.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:30:49.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:49 smithi022 ceph-mon[66341]: failed to encode map e154 with expected crc 2024-03-24T04:30:49.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:49 smithi022 ceph-mon[66341]: failed to encode map e154 with expected crc 2024-03-24T04:30:49.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:49 smithi022 ceph-mon[66341]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T04:30:49.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:49 smithi022 ceph-mon[66341]: pgmap v274: 97 pgs: 2 activating, 1 remapped+peering, 1 remapped, 8 peering, 4 active+clean+remapped, 8 active+undersized, 16 active+undersized+degraded, 57 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 6.8 MiB/s wr, 589 op/s; 137/1638 objects degraded (8.364%); 29/1638 objects misplaced (1.770%) 2024-03-24T04:30:49.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 271841 9.79 MB/sec execute 551 sec latency 1801.367 ms 2024-03-24T04:30:49.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:49 smithi116 ceph-mon[53251]: failed to encode map e154 with expected crc 2024-03-24T04:30:49.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:49 smithi116 ceph-mon[53251]: failed to encode map e154 with expected crc 2024-03-24T04:30:49.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:49 smithi116 ceph-mon[53251]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T04:30:49.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:49 smithi116 ceph-mon[53251]: pgmap v274: 97 pgs: 2 activating, 1 remapped+peering, 1 remapped, 8 peering, 4 active+clean+remapped, 8 active+undersized, 16 active+undersized+degraded, 57 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 6.8 MiB/s wr, 589 op/s; 137/1638 objects degraded (8.364%); 29/1638 objects misplaced (1.770%) 2024-03-24T04:30:50.632 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.633 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: 2.4 starting backfill to osd.6 from (0'0,0'0] MAX to 145'9559 2024-03-24T04:30:50.633 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.633 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: 2.a starting backfill to osd.6 from (0'0,0'0] MAX to 125'8627 2024-03-24T04:30:50.633 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.633 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:50 smithi022 ceph-mon[66341]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T04:30:50.890 INFO:teuthology.orchestra.run.smithi022.stdout: 5 272145 9.78 MB/sec execute 552 sec latency 2111.569 ms 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: 2.4 starting backfill to osd.6 from (0'0,0'0] MAX to 145'9559 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: 2.a starting backfill to osd.6 from (0'0,0'0] MAX to 125'8627 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: failed to encode map e155 with expected crc 2024-03-24T04:30:50.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:50 smithi116 ceph-mon[53251]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T04:30:51.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:51 smithi022 ceph-mon[66341]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:51 smithi022 ceph-mon[66341]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:51 smithi022 ceph-mon[66341]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:51 smithi022 ceph-mon[66341]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T04:30:51.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:51 smithi022 ceph-mon[66341]: pgmap v277: 97 pgs: 2 active+recovery_wait+degraded, 2 activating, 1 remapped+peering, 1 remapped, 9 peering, 1 active+clean+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+undersized+degraded, 68 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s wr, 378 op/s; 834/1632 objects degraded (51.103%); 8/1632 objects misplaced (0.490%); 5.0 MiB/s, 59 keys/s, 8 objects/s recovering 2024-03-24T04:30:51.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 272560 9.79 MB/sec execute 553 sec latency 654.472 ms 2024-03-24T04:30:51.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:51 smithi116 ceph-mon[53251]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:51 smithi116 ceph-mon[53251]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:51 smithi116 ceph-mon[53251]: failed to encode map e156 with expected crc 2024-03-24T04:30:51.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:51 smithi116 ceph-mon[53251]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T04:30:51.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:51 smithi116 ceph-mon[53251]: pgmap v277: 97 pgs: 2 active+recovery_wait+degraded, 2 activating, 1 remapped+peering, 1 remapped, 9 peering, 1 active+clean+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+undersized+degraded, 68 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s wr, 378 op/s; 834/1632 objects degraded (51.103%); 8/1632 objects misplaced (0.490%); 5.0 MiB/s, 59 keys/s, 8 objects/s recovering 2024-03-24T04:30:52.640 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:52 smithi022 ceph-mon[66341]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.640 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:52 smithi022 ceph-mon[66341]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.641 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:52 smithi022 ceph-mon[66341]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.641 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:52 smithi022 ceph-mon[66341]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T04:30:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 272903 9.79 MB/sec execute 554 sec latency 873.671 ms 2024-03-24T04:30:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:52 smithi116 ceph-mon[53251]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:52 smithi116 ceph-mon[53251]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:52 smithi116 ceph-mon[53251]: failed to encode map e157 with expected crc 2024-03-24T04:30:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:52 smithi116 ceph-mon[53251]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T04:30:53.662 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:53 smithi022 ceph-mon[66341]: failed to encode map e158 with expected crc 2024-03-24T04:30:53.662 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:53 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 834/1632 objects degraded (51.103%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:53.662 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:53 smithi022 ceph-mon[66341]: failed to encode map e159 with expected crc 2024-03-24T04:30:53.663 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:53 smithi022 ceph-mon[66341]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T04:30:53.663 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:53 smithi022 ceph-mon[66341]: pgmap v280: 97 pgs: 2 active+recovery_wait+degraded, 2 activating, 1 remapped+peering, 1 remapped, 10 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 77 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.4 MiB/s wr, 366 op/s; 794/1638 objects degraded (48.474%); 4/1638 objects misplaced (0.244%); 6.3 MiB/s, 59 keys/s, 18 objects/s recovering 2024-03-24T04:30:53.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 273329 9.79 MB/sec execute 555 sec latency 38.601 ms 2024-03-24T04:30:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:53 smithi116 ceph-mon[53251]: failed to encode map e158 with expected crc 2024-03-24T04:30:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:53 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 834/1632 objects degraded (51.103%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:53 smithi116 ceph-mon[53251]: failed to encode map e159 with expected crc 2024-03-24T04:30:53.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:53 smithi116 ceph-mon[53251]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T04:30:53.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:53 smithi116 ceph-mon[53251]: pgmap v280: 97 pgs: 2 active+recovery_wait+degraded, 2 activating, 1 remapped+peering, 1 remapped, 10 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 77 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.4 MiB/s wr, 366 op/s; 794/1638 objects degraded (48.474%); 4/1638 objects misplaced (0.244%); 6.3 MiB/s, 59 keys/s, 18 objects/s recovering 2024-03-24T04:30:54.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 273615 9.80 MB/sec execute 556 sec latency 44.840 ms 2024-03-24T04:30:54.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:54 smithi022 ceph-mon[66341]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T04:30:54.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:54 smithi022 ceph-mon[66341]: failed to encode map e160 with expected crc 2024-03-24T04:30:55.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:54 smithi116 ceph-mon[53251]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T04:30:55.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:54 smithi116 ceph-mon[53251]: failed to encode map e160 with expected crc 2024-03-24T04:30:55.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 273930 9.80 MB/sec execute 557 sec latency 36.097 ms 2024-03-24T04:30:55.965 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:55 smithi022 ceph-mon[66341]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T04:30:55.965 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:55 smithi022 ceph-mon[66341]: failed to encode map e161 with expected crc 2024-03-24T04:30:55.965 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:55 smithi022 ceph-mon[66341]: failed to encode map e161 with expected crc 2024-03-24T04:30:55.965 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:55 smithi022 ceph-mon[66341]: pgmap v283: 97 pgs: 1 remapped+peering, 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 91 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 682 op/s; 742/1641 objects degraded (45.216%); 4/1641 objects misplaced (0.244%); 2.5 MiB/s, 0 keys/s, 45 objects/s recovering 2024-03-24T04:30:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:55 smithi116 ceph-mon[53251]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T04:30:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:55 smithi116 ceph-mon[53251]: failed to encode map e161 with expected crc 2024-03-24T04:30:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:55 smithi116 ceph-mon[53251]: failed to encode map e161 with expected crc 2024-03-24T04:30:56.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:55 smithi116 ceph-mon[53251]: pgmap v283: 97 pgs: 1 remapped+peering, 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 91 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 682 op/s; 742/1641 objects degraded (45.216%); 4/1641 objects misplaced (0.244%); 2.5 MiB/s, 0 keys/s, 45 objects/s recovering 2024-03-24T04:30:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 274302 9.80 MB/sec execute 558 sec latency 30.881 ms 2024-03-24T04:30:56.973 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:56 smithi022 ceph-mon[66341]: failed to encode map e162 with expected crc 2024-03-24T04:30:56.973 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:56 smithi022 ceph-mon[66341]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T04:30:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:56 smithi116 ceph-mon[53251]: failed to encode map e162 with expected crc 2024-03-24T04:30:57.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:56 smithi116 ceph-mon[53251]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T04:30:57.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 274545 9.79 MB/sec execute 559 sec latency 47.646 ms 2024-03-24T04:30:57.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:57 smithi022 ceph-mon[66341]: pgmap v285: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.3 MiB/s wr, 481 op/s; 2.2 MiB/s, 2 keys/s, 357 objects/s recovering 2024-03-24T04:30:57.984 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:57 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 742/1641 objects degraded (45.216%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:58.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:57 smithi116 ceph-mon[53251]: pgmap v285: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.3 MiB/s wr, 481 op/s; 2.2 MiB/s, 2 keys/s, 357 objects/s recovering 2024-03-24T04:30:58.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:57 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 742/1641 objects degraded (45.216%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T04:30:58.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 274832 9.79 MB/sec execute 560 sec latency 61.749 ms 2024-03-24T04:30:58.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:58 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 742/1641 objects degraded (45.216%), 2 pgs degraded) 2024-03-24T04:30:58.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:58 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:30:59.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:58 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 742/1641 objects degraded (45.216%), 2 pgs degraded) 2024-03-24T04:30:59.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:58 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:30:59.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 275165 9.78 MB/sec execute 561 sec latency 61.242 ms 2024-03-24T04:31:00.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:30:59 smithi022 ceph-mon[66341]: pgmap v286: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.1 MiB/s wr, 693 op/s; 1.6 MiB/s, 1 keys/s, 257 objects/s recovering 2024-03-24T04:31:00.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:30:59 smithi116 ceph-mon[53251]: pgmap v286: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.1 MiB/s wr, 693 op/s; 1.6 MiB/s, 1 keys/s, 257 objects/s recovering 2024-03-24T04:31:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 275553 9.77 MB/sec execute 562 sec latency 54.172 ms 2024-03-24T04:31:01.307 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:01 smithi022 ceph-mon[66341]: pgmap v287: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3 KiB/s rd, 5.5 MiB/s wr, 509 op/s; 659 KiB/s, 1 keys/s, 204 objects/s recovering 2024-03-24T04:31:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:01 smithi116 ceph-mon[53251]: pgmap v287: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3 KiB/s rd, 5.5 MiB/s wr, 509 op/s; 659 KiB/s, 1 keys/s, 204 objects/s recovering 2024-03-24T04:31:01.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 275822 9.76 MB/sec execute 563 sec latency 44.049 ms 2024-03-24T04:31:02.599 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:02 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:31:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:02 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:31:02.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 276187 9.77 MB/sec execute 564 sec latency 55.259 ms 2024-03-24T04:31:03.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 276644 9.77 MB/sec execute 565 sec latency 36.340 ms 2024-03-24T04:31:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:04 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:31:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:04 smithi116 ceph-mon[53251]: Upgrade: osd.7 is safe to restart 2024-03-24T04:31:04.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:04.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:04 smithi116 ceph-mon[53251]: pgmap v288: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.7 MiB/s wr, 334 op/s; 576 KiB/s, 0 keys/s, 171 objects/s recovering 2024-03-24T04:31:04.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:04.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:04 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T04:31:04.382 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:04 smithi022 ceph-mon[66341]: Upgrade: osd.7 is safe to restart 2024-03-24T04:31:04.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:04.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:04 smithi022 ceph-mon[66341]: pgmap v288: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.7 MiB/s wr, 334 op/s; 576 KiB/s, 0 keys/s, 171 objects/s recovering 2024-03-24T04:31:04.383 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:04.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 276977 9.78 MB/sec execute 566 sec latency 41.092 ms 2024-03-24T04:31:05.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:05 smithi022 ceph-mon[66341]: Upgrade: Updating osd.7 2024-03-24T04:31:05.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:05 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:05.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:05 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:31:05.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:05 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:05.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:05 smithi022 ceph-mon[66341]: Deploying daemon osd.7 on smithi116 2024-03-24T04:31:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:05 smithi116 ceph-mon[53251]: Upgrade: Updating osd.7 2024-03-24T04:31:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:05 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:05.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:05 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T04:31:05.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:05 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:05.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:05 smithi116 ceph-mon[53251]: Deploying daemon osd.7 on smithi116 2024-03-24T04:31:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 277302 9.79 MB/sec execute 567 sec latency 41.413 ms 2024-03-24T04:31:06.193 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:06 smithi022 ceph-mon[66341]: pgmap v289: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 502 op/s; 477 KiB/s, 0 keys/s, 140 objects/s recovering 2024-03-24T04:31:06.347 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:06 smithi116 ceph-mon[53251]: pgmap v289: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 502 op/s; 477 KiB/s, 0 keys/s, 140 objects/s recovering 2024-03-24T04:31:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: 5 277716 9.79 MB/sec execute 568 sec latency 35.128 ms 2024-03-24T04:31:07.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 278010 9.78 MB/sec execute 569 sec latency 52.371 ms 2024-03-24T04:31:07.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:07 smithi116 systemd[1]: Stopping Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:31:08.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:08 smithi022 ceph-mon[66341]: pgmap v290: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 8.7 KiB/s, 121 objects/s recovering 2024-03-24T04:31:08.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:08 smithi022 ceph-mon[66341]: osd.7 marked itself down and dead 2024-03-24T04:31:08.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:08 smithi116 ceph-mon[53251]: pgmap v290: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 8.7 KiB/s, 121 objects/s recovering 2024-03-24T04:31:08.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:08 smithi116 ceph-mon[53251]: osd.7 marked itself down and dead 2024-03-24T04:31:08.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[40405]: 2024-03-24T04:31:08.167+0000 7f71a557f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:31:08.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[40405]: 2024-03-24T04:31:08.167+0000 7f71a557f700 -1 osd.7 162 *** Got signal Terminated *** 2024-03-24T04:31:08.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[40405]: 2024-03-24T04:31:08.167+0000 7f71a557f700 -1 osd.7 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:31:08.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 278082 9.77 MB/sec execute 570 sec latency 728.704 ms 2024-03-24T04:31:09.150 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:08 smithi116 podman[64479]: 2024-03-24 04:31:08.862203243 +0000 UTC m=+0.903339636 container died da6d025024ec1ff836718724518e056daa85d2b82a8d39d7bff2d53a2606aa5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, ceph=True) 2024-03-24T04:31:09.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:09 smithi022 ceph-mon[66341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:31:09.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:09 smithi022 ceph-mon[66341]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T04:31:09.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:09 smithi022 ceph-mon[66341]: osdmap e163: 8 total, 7 up, 8 in 2024-03-24T04:31:09.427 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:09 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:09 smithi116 ceph-mon[53251]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T04:31:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:09 smithi116 ceph-mon[53251]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T04:31:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:09 smithi116 ceph-mon[53251]: osdmap e163: 8 total, 7 up, 8 in 2024-03-24T04:31:09.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:09 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:09.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:09 smithi116 podman[64479]: 2024-03-24 04:31:09.250112007 +0000 UTC m=+1.291248381 container cleanup da6d025024ec1ff836718724518e056daa85d2b82a8d39d7bff2d53a2606aa5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., RELEASE=HEAD, distribution-scope=public, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , ceph=True, version=8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870) 2024-03-24T04:31:09.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:09 smithi116 bash[64479]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7 2024-03-24T04:31:09.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 278244 9.76 MB/sec execute 571 sec latency 1055.804 ms 2024-03-24T04:31:09.904 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:09 smithi116 podman[64496]: 2024-03-24 04:31:09.526751011 +0000 UTC m=+0.658727839 container remove da6d025024ec1ff836718724518e056daa85d2b82a8d39d7bff2d53a2606aa5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.component=centos-stream-container, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754) 2024-03-24T04:31:10.184 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:10 smithi022 ceph-mon[66341]: pgmap v292: 97 pgs: 15 peering, 8 stale+active+clean, 74 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 400 op/s 2024-03-24T04:31:10.185 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:10 smithi022 ceph-mon[66341]: osdmap e164: 8 total, 7 up, 8 in 2024-03-24T04:31:10.185 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:10 smithi022 ceph-mon[66341]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-24T04:31:10.185 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:09 smithi116 podman[64583]: 2024-03-24 04:31:09.807323438 +0000 UTC m=+0.019683967 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:31:10.186 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:09 smithi116 podman[64583]: 2024-03-24 04:31:09.919479261 +0000 UTC m=+0.131839790 container create f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T04:31:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:10 smithi116 ceph-mon[53251]: pgmap v292: 97 pgs: 15 peering, 8 stale+active+clean, 74 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 400 op/s 2024-03-24T04:31:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:10 smithi116 ceph-mon[53251]: osdmap e164: 8 total, 7 up, 8 in 2024-03-24T04:31:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:10 smithi116 ceph-mon[53251]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-24T04:31:10.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:10 smithi116 podman[64583]: 2024-03-24 04:31:10.208309149 +0000 UTC m=+0.420669684 container init f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T04:31:10.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:10 smithi116 podman[64583]: 2024-03-24 04:31:10.211758605 +0000 UTC m=+0.424119141 container start f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T04:31:10.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:10 smithi116 podman[64583]: 2024-03-24 04:31:10.314288821 +0000 UTC m=+0.526649366 container attach f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T04:31:10.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 278619 9.76 MB/sec execute 572 sec latency 63.157 ms 2024-03-24T04:31:10.922 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:10 smithi116 podman[64583]: 2024-03-24 04:31:10.531490325 +0000 UTC m=+0.743850866 container died f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T04:31:11.182 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64609]: 2024-03-24 04:31:11.16914614 +0000 UTC m=+0.631897098 container remove f27e410fdccd6f109e7252f0c238f50a6b54e71bc596fc6e61846535181cfe83 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T04:31:11.450 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:11 smithi022 ceph-mon[66341]: pgmap v294: 97 pgs: 2 active+undersized, 9 active+undersized+degraded, 20 peering, 1 stale+active+clean, 65 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 496 op/s; 59/1692 objects degraded (3.487%) 2024-03-24T04:31:11.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:11 smithi022 ceph-mon[66341]: Health check failed: Degraded data redundancy: 59/1692 objects degraded (3.487%), 9 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:11 smithi116 ceph-mon[53251]: pgmap v294: 97 pgs: 2 active+undersized, 9 active+undersized+degraded, 20 peering, 1 stale+active+clean, 65 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 496 op/s; 59/1692 objects degraded (3.487%) 2024-03-24T04:31:11.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:11 smithi116 ceph-mon[53251]: Health check failed: Degraded data redundancy: 59/1692 objects degraded (3.487%), 9 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:11.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-24T04:31:11.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 systemd[1]: Stopped Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:31:11.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service: Consumed 6min 52.935s CPU time. 2024-03-24T04:31:11.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 systemd[1]: Starting Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:31:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 279042 9.74 MB/sec execute 573 sec latency 43.597 ms 2024-03-24T04:31:11.965 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64710]: 2024-03-24 04:31:11.537859934 +0000 UTC m=+0.019190920 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:31:11.965 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64710]: 2024-03-24 04:31:11.645749979 +0000 UTC m=+0.127080949 container create 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:31:11.965 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64710]: 2024-03-24 04:31:11.865082447 +0000 UTC m=+0.346413428 container init 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:31:11.965 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64710]: 2024-03-24 04:31:11.868602016 +0000 UTC m=+0.349932985 container start 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T04:31:11.966 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:11 smithi116 podman[64710]: 2024-03-24 04:31:11.965439412 +0000 UTC m=+0.446770385 container attach 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T04:31:12.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 bash[64710]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T04:31:12.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T04:31:12.725 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T04:31:12.725 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate[64725]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T04:31:12.725 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:12 smithi116 podman[64710]: 2024-03-24 04:31:12.405126327 +0000 UTC m=+0.886457290 container died 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T04:31:12.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 279384 9.74 MB/sec execute 574 sec latency 38.123 ms 2024-03-24T04:31:13.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 podman[64842]: 2024-03-24 04:31:13.153588229 +0000 UTC m=+0.742744693 container remove 86974a2512122cc2a6e82bfd9b8c3b3216e41ad3d599eaa3a5cf28b67a7da77f (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T04:31:13.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 podman[64904]: 2024-03-24 04:31:13.278523694 +0000 UTC m=+0.019416338 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:31:13.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 podman[64904]: 2024-03-24 04:31:13.396356037 +0000 UTC m=+0.137248646 container create a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T04:31:13.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 279870 9.76 MB/sec execute 575 sec latency 70.713 ms 2024-03-24T04:31:13.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 podman[64904]: 2024-03-24 04:31:13.666399893 +0000 UTC m=+0.407292516 container init a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T04:31:13.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 podman[64904]: 2024-03-24 04:31:13.669678759 +0000 UTC m=+0.410571368 container start a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T04:31:13.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 ceph-osd[64922]: -- 172.21.15.116:0/1679996135 <== mon.1 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639405b71e0 con 0x563940588c00 2024-03-24T04:31:13.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 bash[64904]: a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 2024-03-24T04:31:13.974 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:13 smithi116 systemd[1]: Started Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:31:14.268 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:14 smithi022 ceph-mon[66341]: pgmap v295: 97 pgs: 2 active+undersized, 15 active+undersized+degraded, 20 peering, 60 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.6 MiB/s wr, 327 op/s; 93/1707 objects degraded (5.448%) 2024-03-24T04:31:14.269 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:14 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:14.269 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:14 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:14.269 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:14 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:14.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:31:14.209+0000 7f5c758307c0 -1 Falling back to public interface 2024-03-24T04:31:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:14 smithi116 ceph-mon[53251]: pgmap v295: 97 pgs: 2 active+undersized, 15 active+undersized+degraded, 20 peering, 60 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.6 MiB/s wr, 327 op/s; 93/1707 objects degraded (5.448%) 2024-03-24T04:31:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:14 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:14 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:14.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:14 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:14.782 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:31:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 280345 9.76 MB/sec execute 576 sec latency 38.580 ms 2024-03-24T04:31:15.295 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:15 smithi022 ceph-mon[66341]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:15.295 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:15 smithi022 ceph-mon[66341]: pgmap v296: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 608 op/s; 220/1629 objects degraded (13.505%) 2024-03-24T04:31:15.295 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:15 smithi022 ceph-mon[66341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T04:31:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:15 smithi116 ceph-mon[53251]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:15 smithi116 ceph-mon[53251]: pgmap v296: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 MiB/s wr, 608 op/s; 220/1629 objects degraded (13.505%) 2024-03-24T04:31:15.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:15 smithi116 ceph-mon[53251]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-24T04:31:15.795 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (9m) 2m ago 18m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (5m) 2m ago 17m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (5m) 33s ago 17m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (8m) 2m ago 17m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (12m) 2m ago 12m 37.4M - 17.2.0 e1d6a67b021e c30995920834 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (12m) 33s ago 12m 16.7M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (8m) 2m ago 19m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:31:15.796 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (7m) 33s ago 17m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (6m) 2m ago 19m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (6m) 33s ago 17m 45.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (11m) 2m ago 11m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (11m) 33s ago 11m 88.3M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (8m) 2m ago 17m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 33s ago 17m 18.8M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (4m) 2m ago 16m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (3m) 2m ago 16m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (3m) 2m ago 15m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:31:15.797 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (2m) 2m ago 15m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:31:15.798 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (111s) 33s ago 14m 275M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:31:15.798 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (75s) 33s ago 14m 274M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:31:15.798 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (38s) 33s ago 13m 15.0M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:31:15.798 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 starting - - - 2704M 2024-03-24T04:31:15.798 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (7m) 2m ago 16m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:31:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 280687 9.77 MB/sec execute 577 sec latency 32.594 ms 2024-03-24T04:31:16.362 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:16 smithi022 ceph-mon[66341]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:16.362 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:16 smithi022 ceph-mon[66341]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:16.414 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:16.415 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 11 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:31:16.416 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:31:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:16 smithi116 ceph-mon[53251]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:16 smithi116 ceph-mon[53251]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:16.880 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:31:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr" 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "13/23 daemons upgraded", 2024-03-24T04:31:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T04:31:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:31:16.882 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:31:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 281035 9.77 MB/sec execute 578 sec latency 44.284 ms 2024-03-24T04:31:17.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:17 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1466439122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:17.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:17 smithi022 ceph-mon[66341]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:17.377 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:17 smithi022 ceph-mon[66341]: pgmap v297: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.5 MiB/s wr, 578 op/s; 225/1662 objects degraded (13.538%) 2024-03-24T04:31:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:17 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1466439122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:17 smithi116 ceph-mon[53251]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:17 smithi116 ceph-mon[53251]: pgmap v297: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.5 MiB/s wr, 578 op/s; 225/1662 objects degraded (13.538%) 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 225/1662 objects degraded (13.538%), 29 pgs degraded 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout: osd.7 (root=default,host=smithi116) is down 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 225/1662 objects degraded (13.538%), 29 pgs degraded 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-24T04:31:17.557 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.4 is active+undersized+degraded, acting [1,6] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.a is active+undersized+degraded, acting [1,6] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-24T04:31:17.558 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.8 is active+undersized+degraded, acting [6,1] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.9 is active+undersized+degraded, acting [6,2] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.e is active+undersized+degraded, acting [2,1] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.11 is active+undersized+degraded, acting [1,6] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-24T04:31:17.559 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 3.1b is active+undersized+degraded, acting [0,3] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-24T04:31:17.560 INFO:teuthology.orchestra.run.smithi022.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-24T04:31:17.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 281377 9.77 MB/sec execute 579 sec latency 49.936 ms 2024-03-24T04:31:18.391 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:18 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 225/1662 objects degraded (13.538%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:18.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:18 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/4021953131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:31:18.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:18.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:18 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 225/1662 objects degraded (13.538%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:18 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/4021953131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:31:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:18.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:18.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 281634 9.76 MB/sec execute 580 sec latency 42.158 ms 2024-03-24T04:31:19.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:19.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:19.407 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:19 smithi022 ceph-mon[66341]: pgmap v298: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 451 op/s; 239/1722 objects degraded (13.879%) 2024-03-24T04:31:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:19 smithi116 ceph-mon[53251]: pgmap v298: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 451 op/s; 239/1722 objects degraded (13.879%) 2024-03-24T04:31:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 281854 9.75 MB/sec execute 581 sec latency 41.438 ms 2024-03-24T04:31:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: 5 282175 9.75 MB/sec execute 582 sec latency 93.214 ms 2024-03-24T04:31:20.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:20.950 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:20 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:20.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:20.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:20 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:21 smithi116 ceph-mon[53251]: pgmap v299: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 346 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 516 op/s; 238/1680 objects degraded (14.167%) 2024-03-24T04:31:21.723 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:21 smithi022 ceph-mon[66341]: pgmap v299: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 346 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 516 op/s; 238/1680 objects degraded (14.167%) 2024-03-24T04:31:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 282512 9.74 MB/sec execute 583 sec latency 63.741 ms 2024-03-24T04:31:22.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 282845 9.74 MB/sec execute 584 sec latency 43.017 ms 2024-03-24T04:31:23.016 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:22 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 238/1680 objects degraded (14.167%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:23.045 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:22 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 238/1680 objects degraded (14.167%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:23.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 283252 9.75 MB/sec execute 585 sec latency 55.685 ms 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: pgmap v300: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.7 MiB/s wr, 508 op/s; 230/1656 objects degraded (13.889%) 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T04:31:24.049 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T04:31:24.050 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T04:31:24.050 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T04:31:24.050 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T04:31:24.050 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T04:31:24.050 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T04:31:24.051 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T04:31:24.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:31:24.115+0000 7f5c758307c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: pgmap v300: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.7 MiB/s wr, 508 op/s; 230/1656 objects degraded (13.889%) 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:31:24.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T04:31:24.225 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T04:31:24.226 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T04:31:24.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T04:31:24.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T04:31:24.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T04:31:24.227 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T04:31:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 283688 9.75 MB/sec execute 586 sec latency 55.248 ms 2024-03-24T04:31:25.070 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all osd 2024-03-24T04:31:25.070 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T04:31:25.070 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T04:31:25.071 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T04:31:25.071 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: osdmap e165: 8 total, 7 up, 8 in 2024-03-24T04:31:25.071 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:24 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi022.ekdqmp"]}]: dispatch 2024-03-24T04:31:25.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all osd 2024-03-24T04:31:25.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T04:31:25.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T04:31:25.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T04:31:25.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: osdmap e165: 8 total, 7 up, 8 in 2024-03-24T04:31:25.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:24 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi022.ekdqmp"]}]: dispatch 2024-03-24T04:31:25.895 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284061 9.75 MB/sec execute 587 sec latency 38.870 ms 2024-03-24T04:31:25.973 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:31:25.705+0000 7f5c758307c0 -1 osd.7 162 log_to_monitors true 2024-03-24T04:31:25.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:25 smithi116 ceph-mon[53251]: Upgrade: It appears safe to stop mds.foofs.smithi022.ekdqmp 2024-03-24T04:31:25.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:25 smithi116 ceph-mon[53251]: pgmap v302: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 430 op/s; 217/1575 objects degraded (13.778%) 2024-03-24T04:31:25.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:25 smithi116 ceph-mon[53251]: from='osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:31:25.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:25 smithi116 ceph-mon[53251]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:31:26.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:25 smithi022 ceph-mon[66341]: Upgrade: It appears safe to stop mds.foofs.smithi022.ekdqmp 2024-03-24T04:31:26.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:25 smithi022 ceph-mon[66341]: pgmap v302: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 430 op/s; 217/1575 objects degraded (13.778%) 2024-03-24T04:31:26.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:25 smithi022 ceph-mon[66341]: from='osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:31:26.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:25 smithi022 ceph-mon[66341]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T04:31:26.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:31:26.376+0000 7f5c62c80700 -1 osd.7 162 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T04:31:26.894 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284349 9.76 MB/sec execute 588 sec latency 152.631 ms 2024-03-24T04:31:27.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:26 smithi022 ceph-mon[66341]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T04:31:27.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:26 smithi022 ceph-mon[66341]: osdmap e166: 8 total, 7 up, 8 in 2024-03-24T04:31:27.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:26 smithi022 ceph-mon[66341]: from='osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:31:27.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:26 smithi022 ceph-mon[66341]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:31:27.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:26 smithi022 ceph-mon[66341]: from='osd.7 ' entity='osd.7' 2024-03-24T04:31:27.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-mon[53251]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T04:31:27.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-mon[53251]: osdmap e166: 8 total, 7 up, 8 in 2024-03-24T04:31:27.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-mon[53251]: from='osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:31:27.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-mon[53251]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T04:31:27.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:26 smithi116 ceph-mon[53251]: from='osd.7 ' entity='osd.7' 2024-03-24T04:31:27.895 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284483 9.75 MB/sec execute 589 sec latency 531.347 ms 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: pgmap v304: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 666 op/s; 224/1593 objects degraded (14.062%) 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:31:28.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920] boot 2024-03-24T04:31:28.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T04:31:28.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:31:28.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 217/1575 objects degraded (13.778%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:28.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:27 smithi116 ceph-mon[53251]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T04:31:28.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: pgmap v304: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 60 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 666 op/s; 224/1593 objects degraded (14.062%) 2024-03-24T04:31:28.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:28.226 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi022.ekdqmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: osd.7 [v2:172.21.15.116:6824/3243705920,v1:172.21.15.116:6825/3243705920] boot 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 217/1575 objects degraded (13.778%), 29 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:28.227 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:27 smithi022 ceph-mon[66341]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T04:31:28.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284483 9.73 MB/sec execute 590 sec latency 1534.656 ms 2024-03-24T04:31:29.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:28 smithi022 ceph-mon[66341]: Upgrade: Updating mds.foofs.smithi022.ekdqmp 2024-03-24T04:31:29.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:28 smithi022 ceph-mon[66341]: Deploying daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:31:29.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:28 smithi022 ceph-mon[66341]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T04:31:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:28 smithi116 ceph-mon[53251]: Upgrade: Updating mds.foofs.smithi022.ekdqmp 2024-03-24T04:31:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:28 smithi116 ceph-mon[53251]: Deploying daemon mds.foofs.smithi022.ekdqmp on smithi022 2024-03-24T04:31:29.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:28 smithi116 ceph-mon[53251]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T04:31:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284680 9.72 MB/sec execute 591 sec latency 1944.267 ms 2024-03-24T04:31:30.126 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:29 smithi022 ceph-mon[66341]: pgmap v308: 97 pgs: 4 remapped+peering, 3 active+undersized, 21 active+undersized+degraded, 69 active+clean; 344 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.9 MiB/s wr, 522 op/s; 148/1641 objects degraded (9.019%); 33 KiB/s, 5 objects/s recovering 2024-03-24T04:31:30.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:29 smithi116 ceph-mon[53251]: pgmap v308: 97 pgs: 4 remapped+peering, 3 active+undersized, 21 active+undersized+degraded, 69 active+clean; 344 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.9 MiB/s wr, 522 op/s; 148/1641 objects degraded (9.019%); 33 KiB/s, 5 objects/s recovering 2024-03-24T04:31:30.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[66334]: 2024-03-24T04:31:30.126+0000 7f7d9c519700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:30.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284689 9.71 MB/sec execute 592 sec latency 1006.505 ms 2024-03-24T04:31:31.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T04:31:31.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: Standby daemon mds.foofs.smithi116.kvgpsf assigned to filesystem foofs as rank 0 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T04:31:31.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:30 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:replay} 2024-03-24T04:31:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T04:31:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T04:31:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T04:31:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: Standby daemon mds.foofs.smithi116.kvgpsf assigned to filesystem foofs as rank 0 2024-03-24T04:31:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T04:31:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:31:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T04:31:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:30 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:replay} 2024-03-24T04:31:31.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284689 9.69 MB/sec execute 593 sec latency 2006.536 ms 2024-03-24T04:31:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T04:31:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: pgmap v312: 97 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 2 remapped+peering, 5 active+undersized+degraded, 88 active+clean; 342 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.3 MiB/s wr, 169 op/s; 31/1653 objects degraded (1.875%); 2.2 MiB/s, 56 objects/s recovering 2024-03-24T04:31:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T04:31:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:reconnect 2024-03-24T04:31:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.22:6834/303924342,v1:172.21.15.22:6835/303924342] up:boot 2024-03-24T04:31:32.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:reconnect} 1 up:standby 2024-03-24T04:31:32.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:32 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:31:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T04:31:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: pgmap v312: 97 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 2 remapped+peering, 5 active+undersized+degraded, 88 active+clean; 342 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 4.3 MiB/s wr, 169 op/s; 31/1653 objects degraded (1.875%); 2.2 MiB/s, 56 objects/s recovering 2024-03-24T04:31:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T04:31:32.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:reconnect 2024-03-24T04:31:32.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.22:6834/303924342,v1:172.21.15.22:6835/303924342] up:boot 2024-03-24T04:31:32.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:reconnect} 1 up:standby 2024-03-24T04:31:32.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:32 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:31:32.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284689 9.67 MB/sec execute 594 sec latency 3006.562 ms 2024-03-24T04:31:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:33 smithi116 ceph-mon[53251]: reconnect by client.24339 172.21.15.116:0/3505700020 after 0.000999984 2024-03-24T04:31:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:33 smithi116 ceph-mon[53251]: reconnect by client.14564 172.21.15.22:0/1620835732 after 0.000999984 2024-03-24T04:31:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:33 smithi116 ceph-mon[53251]: Health check update: Degraded data redundancy: 31/1653 objects degraded (1.875%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:33 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:rejoin 2024-03-24T04:31:33.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:33 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:rejoin} 1 up:standby 2024-03-24T04:31:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:33 smithi022 ceph-mon[66341]: reconnect by client.24339 172.21.15.116:0/3505700020 after 0.000999984 2024-03-24T04:31:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:33 smithi022 ceph-mon[66341]: reconnect by client.14564 172.21.15.22:0/1620835732 after 0.000999984 2024-03-24T04:31:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:33 smithi022 ceph-mon[66341]: Health check update: Degraded data redundancy: 31/1653 objects degraded (1.875%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T04:31:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:33 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:rejoin 2024-03-24T04:31:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:33 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:rejoin} 1 up:standby 2024-03-24T04:31:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284689 9.66 MB/sec execute 595 sec latency 4006.586 ms 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: pgmap v313: 97 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 2 remapped+peering, 93 active+clean; 346 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.2 MiB/s wr, 179 op/s; 1/1656 objects degraded (0.060%); 3.5 MiB/s, 50 objects/s recovering 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:clientreplay 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:clientreplay} 1 up:standby 2024-03-24T04:31:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:34 smithi116 ceph-mon[53251]: daemon mds.foofs.smithi116.kvgpsf is now active in filesystem foofs as rank 0 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: pgmap v313: 97 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 2 remapped+peering, 93 active+clean; 346 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.2 MiB/s wr, 179 op/s; 1/1656 objects degraded (0.060%); 3.5 MiB/s, 50 objects/s recovering 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:clientreplay 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi116.kvgpsf=up:clientreplay} 1 up:standby 2024-03-24T04:31:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:34 smithi022 ceph-mon[66341]: daemon mds.foofs.smithi116.kvgpsf is now active in filesystem foofs as rank 0 2024-03-24T04:31:34.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 284689 9.64 MB/sec execute 596 sec latency 5006.633 ms 2024-03-24T04:31:35.159 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:35 smithi022 ceph-mon[66341]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T04:31:35.159 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:35 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:active 2024-03-24T04:31:35.159 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:35 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi116.kvgpsf=up:active} 1 up:standby 2024-03-24T04:31:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:35 smithi116 ceph-mon[53251]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T04:31:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:35 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.116:6832/258618097,v1:172.21.15.116:6833/258618097] up:active 2024-03-24T04:31:35.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:35 smithi116 ceph-mon[53251]: fsmap foofs:1 {0=foofs.smithi116.kvgpsf=up:active} 1 up:standby 2024-03-24T04:31:35.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 285028 9.64 MB/sec execute 597 sec latency 5029.344 ms 2024-03-24T04:31:36.173 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:36 smithi022 ceph-mon[66341]: pgmap v314: 97 pgs: 1 peering, 96 active+clean; 346 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 1.8 MiB/s wr, 147 op/s; 3.4 MiB/s, 45 objects/s recovering 2024-03-24T04:31:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:36 smithi116 ceph-mon[53251]: pgmap v314: 97 pgs: 1 peering, 96 active+clean; 346 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 1.8 MiB/s wr, 147 op/s; 3.4 MiB/s, 45 objects/s recovering 2024-03-24T04:31:36.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 285258 9.63 MB/sec execute 598 sec latency 34.771 ms 2024-03-24T04:31:37.222 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:37 smithi022 ceph-mon[66341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1656 objects degraded (0.060%), 1 pg degraded) 2024-03-24T04:31:37.223 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:37 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:31:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:37 smithi116 ceph-mon[53251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1656 objects degraded (0.060%), 1 pg degraded) 2024-03-24T04:31:37.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:37 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:31:37.898 INFO:teuthology.orchestra.run.smithi022.stdout: 5 285575 9.63 MB/sec execute 599 sec latency 62.199 ms 2024-03-24T04:31:38.229 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:38 smithi022 ceph-mon[66341]: pgmap v315: 97 pgs: 97 active+clean; 347 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 1.5 MiB/s wr, 143 op/s; 2.9 MiB/s, 39 objects/s recovering 2024-03-24T04:31:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:38 smithi116 ceph-mon[53251]: pgmap v315: 97 pgs: 97 active+clean; 347 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 1.5 MiB/s wr, 143 op/s; 2.9 MiB/s, 39 objects/s recovering 2024-03-24T04:31:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: 5 cleanup 600 sec 2024-03-24T04:31:39.020 INFO:teuthology.orchestra.run.smithi022.stdout:[285785] unlink './clients/client3/~dmtmp/PARADOX/.nfs00000100000002cf00000005' failed - Device or resource busy 2024-03-24T04:31:39.021 INFO:teuthology.orchestra.run.smithi022.stdout:[285785] unlink './clients/client3/~dmtmp/PARADOX/.nfs00000100000002ca00000001' failed - Device or resource busy 2024-03-24T04:31:39.021 INFO:teuthology.orchestra.run.smithi022.stdout:[285785] unlink './clients/client3/~dmtmp/PARADOX/.nfs00000100000002d30000000d' failed - Device or resource busy 2024-03-24T04:31:39.028 INFO:teuthology.orchestra.run.smithi022.stdout:[285785] unlink './clients/client3/~dmtmp/PARADOX/.nfs00000100000002c50000000b' failed - Device or resource busy 2024-03-24T04:31:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: 0 cleanup 600 sec 2024-03-24T04:31:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:31:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: Operation Count AvgLat MaxLat 2024-03-24T04:31:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: ---------------------------------------- 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: NTCreateX 185794 4.656 5029.339 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Close 136340 0.141 6.292 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Rename 7857 10.425 2205.235 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Unlink 37624 7.069 5022.008 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Qpathinfo 168378 1.165 1035.521 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Qfileinfo 29306 0.008 2.662 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Qfsinfo 30850 0.686 34.704 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Sfileinfo 15139 12.776 2325.848 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: Find 64993 1.011 13.860 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: WriteX 91623 14.020 5016.315 2024-03-24T04:31:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: ReadX 290655 0.010 6.395 2024-03-24T04:31:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: LockX 602 0.421 4.455 2024-03-24T04:31:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: UnlockX 602 0.368 6.646 2024-03-24T04:31:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: Flush 13039 0.002 0.098 2024-03-24T04:31:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:31:39.104 INFO:teuthology.orchestra.run.smithi022.stdout:Throughput 9.63013 MB/sec 5 clients 5 procs max_latency=5029.344 ms 2024-03-24T04:31:39.106 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-03-24T04:31:39.143 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-03-24T04:31:39.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-03-24T04:31:39.302 INFO:teuthology.orchestra.run.smithi022.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T04:31:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:31:39.303 INFO:teuthology.orchestra.run.smithi022.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-03-24T04:31:39.303 INFO:teuthology.orchestra.run.smithi022.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-24T04:31:39.403 INFO:teuthology.orchestra.run.smithi022.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-24T04:31:39.404 INFO:teuthology.orchestra.run.smithi022.stdout:releasing clients 2024-03-24T04:31:40.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:40 smithi022 ceph-mon[66341]: pgmap v316: 97 pgs: 97 active+clean; 349 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 2.0 MiB/s wr, 257 op/s; 2.4 MiB/s, 32 objects/s recovering 2024-03-24T04:31:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:40 smithi116 ceph-mon[53251]: pgmap v316: 97 pgs: 97 active+clean; 349 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 2.0 MiB/s wr, 257 op/s; 2.4 MiB/s, 32 objects/s recovering 2024-03-24T04:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: 5 381 25.23 MB/sec execute 1 sec latency 117.205 ms 2024-03-24T04:31:42.397 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:42 smithi022 ceph-mon[66341]: pgmap v317: 97 pgs: 97 active+clean; 352 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 4.1 MiB/s wr, 269 op/s; 1.5 MiB/s, 14 objects/s recovering 2024-03-24T04:31:42.404 INFO:teuthology.orchestra.run.smithi022.stdout: 5 544 26.46 MB/sec execute 2 sec latency 39.214 ms 2024-03-24T04:31:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:42 smithi116 ceph-mon[53251]: pgmap v317: 97 pgs: 97 active+clean; 352 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 4.1 MiB/s wr, 269 op/s; 1.5 MiB/s, 14 objects/s recovering 2024-03-24T04:31:43.404 INFO:teuthology.orchestra.run.smithi022.stdout: 5 794 21.57 MB/sec execute 3 sec latency 53.611 ms 2024-03-24T04:31:43.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:43 smithi022 ceph-mon[66341]: daemon mds.foofs.smithi022.ekdqmp restarted 2024-03-24T04:31:43.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:43.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:43 smithi022 ceph-mon[66341]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T04:31:43.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:43.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:43 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:43 smithi116 ceph-mon[53251]: daemon mds.foofs.smithi022.ekdqmp restarted 2024-03-24T04:31:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:43.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:43 smithi116 ceph-mon[53251]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T04:31:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:43.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:43 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:44.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:44 smithi022 ceph-mon[66341]: pgmap v319: 97 pgs: 97 active+clean; 345 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 4.8 MiB/s wr, 326 op/s; 677 KiB/s, 8 objects/s recovering 2024-03-24T04:31:44.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:44 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:boot 2024-03-24T04:31:44.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:44 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi116.kvgpsf=up:active} 1 up:standby 2024-03-24T04:31:44.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:31:44.202 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:44 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: 5 1078 18.01 MB/sec execute 4 sec latency 53.982 ms 2024-03-24T04:31:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:44 smithi116 ceph-mon[53251]: pgmap v319: 97 pgs: 97 active+clean; 345 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 4.8 MiB/s wr, 326 op/s; 677 KiB/s, 8 objects/s recovering 2024-03-24T04:31:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:44 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:boot 2024-03-24T04:31:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:44 smithi116 ceph-mon[53251]: fsmap foofs:1 {0=foofs.smithi116.kvgpsf=up:active} 1 up:standby 2024-03-24T04:31:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:44 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:31:44.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:44 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:45.404 INFO:teuthology.orchestra.run.smithi022.stdout: 5 cleanup 5 sec 2024-03-24T04:31:45.849 INFO:teuthology.orchestra.run.smithi022.stdout: 0 cleanup 5 sec 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: Operation Count AvgLat MaxLat 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: ---------------------------------------- 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: NTCreateX 830 9.223 69.461 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: Close 749 6.737 47.151 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: Rename 50 16.136 49.162 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: Unlink 87 4.980 20.616 2024-03-24T04:31:45.850 INFO:teuthology.orchestra.run.smithi022.stdout: Qpathinfo 492 2.208 30.037 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: Qfileinfo 353 15.511 54.725 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: Qfsinfo 165 0.627 7.209 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: Sfileinfo 170 14.315 35.937 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: Find 238 0.898 2.868 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: WriteX 1637 0.025 0.232 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: ReadX 1123 0.016 0.451 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: Flush 120 14.514 26.494 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:31:45.851 INFO:teuthology.orchestra.run.smithi022.stdout:Throughput 18.0068 MB/sec 5 clients 5 procs max_latency=117.205 ms 2024-03-24T04:31:45.853 DEBUG:teuthology.parallel:result is None 2024-03-24T04:31:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:46 smithi116 ceph-mon[53251]: pgmap v320: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 460 op/s; 120 KiB/s, 3 objects/s recovering 2024-03-24T04:31:46.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:46 smithi022 ceph-mon[66341]: pgmap v320: 97 pgs: 97 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 460 op/s; 120 KiB/s, 3 objects/s recovering 2024-03-24T04:31:47.922 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:31:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:48 smithi116 ceph-mon[53251]: pgmap v321: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-24T04:31:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:48.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:48 smithi022 ceph-mon[66341]: pgmap v321: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-24T04:31:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (9m) 2m ago 18m 24.4M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (5m) 2m ago 18m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (5m) 30s ago 17m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (9m) 2m ago 18m 52.0M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 starting - - - - 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (12m) 30s ago 12m 16.9M - 17.2.0 e1d6a67b021e d2c10121ebb1 2024-03-24T04:31:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (8m) 2m ago 19m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (7m) 30s ago 17m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (7m) 2m ago 20m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (6m) 30s ago 17m 44.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (12m) 2m ago 12m 275M - 4.0 e1d6a67b021e 21acca2fc979 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (12m) 30s ago 12m 88.3M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (9m) 2m ago 18m 18.5M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (8m) 30s ago 17m 24.8M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 2m ago 17m 347M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (4m) 2m ago 16m 301M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (3m) 2m ago 16m 239M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (3m) 2m ago 15m 12.6M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (2m) 30s ago 15m 302M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (108s) 30s ago 14m 290M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:31:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (70s) 30s ago 14m 333M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:31:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (35s) 30s ago 14m 14.7M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:31:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (7m) 2m ago 17m 68.6M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:31:49.201 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:31:49.202 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:31:49.203 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T04:31:49.203 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 13 2024-03-24T04:31:49.203 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:31:49.203 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:31:49.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:49 smithi116 ceph-mon[53251]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:49.551 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:49 smithi022 ceph-mon[66341]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "osd" 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "14/23 daemons upgraded", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading mds daemons", 2024-03-24T04:31:49.559 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:31:49.560 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:31:50.011 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:31:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:50 smithi116 ceph-mon[53251]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:50 smithi116 ceph-mon[53251]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:50 smithi116 ceph-mon[53251]: pgmap v322: 97 pgs: 97 active+clean; 313 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 421 op/s 2024-03-24T04:31:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:50 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/688324642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:50.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:50 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3144084318' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:31:50.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:50 smithi022 ceph-mon[66341]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:50 smithi022 ceph-mon[66341]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:50 smithi022 ceph-mon[66341]: pgmap v322: 97 pgs: 97 active+clean; 313 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 421 op/s 2024-03-24T04:31:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:50 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/688324642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:50.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:50 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3144084318' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:31:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:51 smithi116 ceph-mon[53251]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:51.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:51 smithi022 ceph-mon[66341]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:31:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:51.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:52 smithi116 ceph-mon[53251]: pgmap v323: 97 pgs: 97 active+clean; 258 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 9.9 MiB/s wr, 385 op/s 2024-03-24T04:31:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:52 smithi022 ceph-mon[66341]: pgmap v323: 97 pgs: 97 active+clean; 258 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 9.9 MiB/s wr, 385 op/s 2024-03-24T04:31:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:54.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:53 smithi022 ceph-mon[66341]: pgmap v324: 97 pgs: 97 active+clean; 244 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 8.5 MiB/s wr, 269 op/s 2024-03-24T04:31:54.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:53 smithi116 ceph-mon[53251]: pgmap v324: 97 pgs: 97 active+clean; 244 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 8.5 MiB/s wr, 269 op/s 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: pgmap v325: 97 pgs: 97 active+clean; 231 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 7.4 MiB/s wr, 236 op/s 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:31:56.331 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.332 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi116.kvgpsf"]}]: dispatch 2024-03-24T04:31:56.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: pgmap v325: 97 pgs: 97 active+clean; 231 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 7.4 MiB/s wr, 236 op/s 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:31:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi116.kvgpsf"]}]: dispatch 2024-03-24T04:31:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:57 smithi116 ceph-mon[53251]: Detected new or changed devices on smithi022 2024-03-24T04:31:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:57 smithi116 ceph-mon[53251]: Upgrade: It appears safe to stop mds.foofs.smithi116.kvgpsf 2024-03-24T04:31:57.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:57.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:31:57.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:57.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:57 smithi022 ceph-mon[66341]: Detected new or changed devices on smithi022 2024-03-24T04:31:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:57 smithi022 ceph-mon[66341]: Upgrade: It appears safe to stop mds.foofs.smithi116.kvgpsf 2024-03-24T04:31:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:31:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi116.kvgpsf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T04:31:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:31:58.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:58 smithi116 ceph-mon[53251]: Upgrade: Updating mds.foofs.smithi116.kvgpsf 2024-03-24T04:31:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:58 smithi116 ceph-mon[53251]: Deploying daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:31:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:58 smithi116 ceph-mon[53251]: pgmap v326: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-24T04:31:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:58 smithi022 ceph-mon[66341]: Upgrade: Updating mds.foofs.smithi116.kvgpsf 2024-03-24T04:31:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:58 smithi022 ceph-mon[66341]: Deploying daemon mds.foofs.smithi116.kvgpsf on smithi116 2024-03-24T04:31:58.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:58 smithi022 ceph-mon[66341]: pgmap v326: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-24T04:31:59.474 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:31:59.089+0000 7f998742e700 -1 bdev(0x55752507e000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-03-24T04:31:59.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: Standby daemon mds.foofs.smithi022.ekdqmp assigned to filesystem foofs as rank 0 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T04:31:59.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:31:59 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:replay} 2024-03-24T04:31:59.547 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:31:59.091+0000 7fd7dec59700 -1 bdev(0x55643da9a000 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 7 2024-03-24T04:31:59.548 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:31:59.091+0000 7fd7e045c700 -1 bdev(0x55643da9a000 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 12 2024-03-24T04:31:59.548 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:31:59.089+0000 7f1bb54cb700 -1 bdev(0x55f91210a000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-03-24T04:31:59.548 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:31:59.091+0000 7f1bb8cd2700 -1 bdev(0x55f91210a000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-03-24T04:31:59.548 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:31:59.091+0000 7f1bb9cd4700 -1 bdev(0x55f91210a000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[66334]: 2024-03-24T04:31:59.070+0000 7f7d9c519700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: Standby daemon mds.foofs.smithi022.ekdqmp assigned to filesystem foofs as rank 0 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T04:31:59.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-24T04:31:59.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:31:59 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:replay} 2024-03-24T04:32:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:00 smithi116 ceph-mon[53251]: pgmap v328: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 186 KiB/s wr, 31 op/s 2024-03-24T04:32:00.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:00 smithi022 ceph-mon[66341]: pgmap v328: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 186 KiB/s wr, 31 op/s 2024-03-24T04:32:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:01 smithi116 ceph-mon[53251]: reconnect by client.14564 172.21.15.22:0/1620835732 after 0 2024-03-24T04:32:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:01 smithi116 ceph-mon[53251]: reconnect by client.24339 172.21.15.116:0/3505700020 after 0 2024-03-24T04:32:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:01 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:reconnect 2024-03-24T04:32:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:01 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:reconnect} 2024-03-24T04:32:01.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:01 smithi116 ceph-mon[53251]: pgmap v329: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-03-24T04:32:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:01 smithi022 ceph-mon[66341]: reconnect by client.14564 172.21.15.22:0/1620835732 after 0 2024-03-24T04:32:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:01 smithi022 ceph-mon[66341]: reconnect by client.24339 172.21.15.116:0/3505700020 after 0 2024-03-24T04:32:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:01 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:reconnect 2024-03-24T04:32:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:01 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:reconnect} 2024-03-24T04:32:01.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:01 smithi022 ceph-mon[66341]: pgmap v329: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-03-24T04:32:02.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:02 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:rejoin 2024-03-24T04:32:02.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:02 smithi022 ceph-mon[66341]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:rejoin} 2024-03-24T04:32:02.550 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:02 smithi022 ceph-mon[66341]: daemon mds.foofs.smithi022.ekdqmp is now active in filesystem foofs as rank 0 2024-03-24T04:32:02.621 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:02 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:rejoin 2024-03-24T04:32:02.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:02 smithi116 ceph-mon[53251]: fsmap foofs:1/1 {0=foofs.smithi022.ekdqmp=up:rejoin} 2024-03-24T04:32:02.622 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:02 smithi116 ceph-mon[53251]: daemon mds.foofs.smithi022.ekdqmp is now active in filesystem foofs as rank 0 2024-03-24T04:32:03.525 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T04:32:03.526 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:active 2024-03-24T04:32:03.526 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 2024-03-24T04:32:03.526 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:03.526 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:03.526 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:03 smithi116 ceph-mon[53251]: pgmap v330: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479] up:active 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:03 smithi022 ceph-mon[66341]: pgmap v330: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T04:32:06.397 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:06 smithi116 ceph-mon[53251]: pgmap v331: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T04:32:06.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:06 smithi022 ceph-mon[66341]: pgmap v331: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: pgmap v332: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: Cluster is now healthy 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: mds.? [v2:172.21.15.116:6832/1663772929,v1:172.21.15.116:6833/1663772929] up:boot 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:32:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:07 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:32:08.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:08.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:08.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:08.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: pgmap v332: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T04:32:08.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-24T04:32:08.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: Cluster is now healthy 2024-03-24T04:32:08.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: mds.? [v2:172.21.15.116:6832/1663772929,v1:172.21.15.116:6833/1663772929] up:boot 2024-03-24T04:32:08.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: fsmap foofs:1 {0=foofs.smithi022.ekdqmp=up:active} 1 up:standby 2024-03-24T04:32:08.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:07 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:32:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:10 smithi116 ceph-mon[53251]: pgmap v333: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 203 B/s wr, 11 op/s 2024-03-24T04:32:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:10 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:10.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:10 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:10.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:10 smithi022 ceph-mon[66341]: pgmap v333: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 203 B/s wr, 11 op/s 2024-03-24T04:32:10.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:10 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:10.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:10 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:12.201 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:12.201 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:12.201 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:11 smithi116 ceph-mon[53251]: pgmap v334: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-24T04:32:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:11 smithi022 ceph-mon[66341]: pgmap v334: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-24T04:32:13.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:13 smithi022 ceph-mon[66341]: pgmap v335: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-24T04:32:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:13 smithi116 ceph-mon[53251]: pgmap v335: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: Detected new or changed devices on smithi116 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:32:15.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi022.ekdqmp"}]': finished 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi116.kvgpsf"}]': finished 2024-03-24T04:32:15.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:15 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: Detected new or changed devices on smithi116 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:15.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi022.ekdqmp"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi022.ekdqmp"}]': finished 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi116.kvgpsf"}]: dispatch 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi116.kvgpsf"}]': finished 2024-03-24T04:32:15.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:15.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:15 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:16.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all mds 2024-03-24T04:32:16.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all rgw 2024-03-24T04:32:16.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T04:32:16.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T04:32:16.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all iscsi 2024-03-24T04:32:16.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:16 smithi022 ceph-mon[66341]: pgmap v336: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 7.1 KiB/s wr, 8 op/s 2024-03-24T04:32:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all mds 2024-03-24T04:32:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all rgw 2024-03-24T04:32:16.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T04:32:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T04:32:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all iscsi 2024-03-24T04:32:16.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:16 smithi116 ceph-mon[53251]: pgmap v336: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 7.1 KiB/s wr, 8 op/s 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Upgrade: Updating nfs.foo.0.0.smithi022.iycgyn (1/2) 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:32:17.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Rados config object exists: conf-nfs.foo 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn-rgw 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Bind address in nfs.foo.0.0.smithi022.iycgyn's ganesha conf is defaulting to empty 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: Deploying daemon nfs.foo.0.0.smithi022.iycgyn on smithi022 2024-03-24T04:32:17.549 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:17 smithi022 ceph-mon[66341]: pgmap v337: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.0 KiB/s wr, 4 op/s 2024-03-24T04:32:17.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Upgrade: Updating nfs.foo.0.0.smithi022.iycgyn (1/2) 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Rados config object exists: conf-nfs.foo 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Creating key for client.nfs.foo.0.0.smithi022.iycgyn-rgw 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi022.iycgyn-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Bind address in nfs.foo.0.0.smithi022.iycgyn's ganesha conf is defaulting to empty 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:17.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: Deploying daemon nfs.foo.0.0.smithi022.iycgyn on smithi022 2024-03-24T04:32:17.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:17 smithi116 ceph-mon[53251]: pgmap v337: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.0 KiB/s wr, 4 op/s 2024-03-24T04:32:19.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:19.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:19 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:19.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:19 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:20.385 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:32:20.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:20 smithi116 ceph-mon[53251]: pgmap v338: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-24T04:32:20.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:20 smithi022 ceph-mon[66341]: pgmap v338: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (10m) 30s ago 19m 24.9M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (6m) 30s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (6m) 11s ago 18m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (9m) 30s ago 18m 54.7M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (39s) 30s ago 13m 12.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 1d0af2aadefd 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (14s) 11s ago 13m 13.6M - 19.0.0-1667-gdb0330b1 4f325f8440e9 9d19bfdc5f25 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (9m) 30s ago 20m 512M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (8m) 11s ago 18m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:32:21.100 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (7m) 30s ago 20m 57.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (7m) 11s ago 18m 44.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 starting - - - - 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (12m) 11s ago 12m 88.4M - 4.0 e1d6a67b021e 85780e4faac3 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (9m) 30s ago 18m 16.0M - 1.3.1 1dbe0e931976 0b9659fba86a 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (9m) 11s ago 18m 19.1M - 1.3.1 1dbe0e931976 8fb645b15236 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 30s ago 17m 440M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (5m) 30s ago 17m 471M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 30s ago 16m 373M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (3m) 30s ago 16m 402M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (2m) 11s ago 15m 330M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (2m) 11s ago 15m 325M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (103s) 11s ago 15m 383M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (67s) 11s ago 14m 283M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:32:21.101 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (8m) 30s ago 17m 70.9M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:32:21.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:21 smithi116 ceph-mon[53251]: Evicting (and blocklisting) client session 14564 (172.21.15.22:0/1620835732) 2024-03-24T04:32:21.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:21 smithi116 ceph-mon[53251]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.22:0/1620835732"}]: dispatch 2024-03-24T04:32:21.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:21 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:21.471 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:21 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:32:21.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:32:21.527 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:32:21.527 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:21 smithi022 ceph-mon[66341]: Evicting (and blocklisting) client session 14564 (172.21.15.22:0/1620835732) 2024-03-24T04:32:21.527 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:21 smithi022 ceph-mon[66341]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.22:0/1620835732"}]: dispatch 2024-03-24T04:32:21.527 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:21 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:21.527 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:21 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "mds", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "osd" 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "16/23 daemons upgraded", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading nfs daemons", 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:32:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:32:22.300 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:32:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: pgmap v339: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.22:0/1620835732"}]': finished 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2904160293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:32:22.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:32:22.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:22.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: pgmap v339: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.22:0/1620835732"}]': finished 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2904160293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-24T04:32:22.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-24T04:32:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: Upgrade: Updating nfs.foo.1.0.smithi116.wxmbpt (2/2) 2024-03-24T04:32:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt 2024-03-24T04:32:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T04:32:23.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:32:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:32:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:32:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:23.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:23 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1001575565' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:32:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: Upgrade: Updating nfs.foo.1.0.smithi116.wxmbpt (2/2) 2024-03-24T04:32:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt 2024-03-24T04:32:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi116.wxmbpt-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:23.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:23 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1001575565' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:32:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:24 smithi116 ceph-mon[53251]: Rados config object exists: conf-nfs.foo 2024-03-24T04:32:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:24 smithi116 ceph-mon[53251]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt-rgw 2024-03-24T04:32:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:24 smithi116 ceph-mon[53251]: Bind address in nfs.foo.1.0.smithi116.wxmbpt's ganesha conf is defaulting to empty 2024-03-24T04:32:24.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:24 smithi116 ceph-mon[53251]: Deploying daemon nfs.foo.1.0.smithi116.wxmbpt on smithi116 2024-03-24T04:32:24.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:24 smithi116 ceph-mon[53251]: pgmap v341: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T04:32:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:24 smithi022 ceph-mon[66341]: Rados config object exists: conf-nfs.foo 2024-03-24T04:32:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:24 smithi022 ceph-mon[66341]: Creating key for client.nfs.foo.1.0.smithi116.wxmbpt-rgw 2024-03-24T04:32:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:24 smithi022 ceph-mon[66341]: Bind address in nfs.foo.1.0.smithi116.wxmbpt's ganesha conf is defaulting to empty 2024-03-24T04:32:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:24 smithi022 ceph-mon[66341]: Deploying daemon nfs.foo.1.0.smithi116.wxmbpt on smithi116 2024-03-24T04:32:24.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:24 smithi022 ceph-mon[66341]: pgmap v341: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T04:32:26.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:26 smithi022 ceph-mon[66341]: pgmap v342: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-24T04:32:26.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:26.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:26.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:26 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:26 smithi116 ceph-mon[53251]: pgmap v342: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-24T04:32:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:26.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:26 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:28.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:28 smithi116 ceph-mon[53251]: pgmap v343: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-24T04:32:28.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:28 smithi022 ceph-mon[66341]: pgmap v343: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-24T04:32:30.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:30.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:30.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:30 smithi022 ceph-mon[66341]: Evicting (and blocklisting) client session 24339 (172.21.15.116:0/3505700020) 2024-03-24T04:32:30.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:30 smithi022 ceph-mon[66341]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.116:0/3505700020"}]: dispatch 2024-03-24T04:32:30.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:30 smithi022 ceph-mon[66341]: pgmap v344: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-24T04:32:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:30 smithi116 ceph-mon[53251]: Evicting (and blocklisting) client session 24339 (172.21.15.116:0/3505700020) 2024-03-24T04:32:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:30 smithi116 ceph-mon[53251]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.116:0/3505700020"}]: dispatch 2024-03-24T04:32:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:30 smithi116 ceph-mon[53251]: pgmap v344: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-24T04:32:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:31 smithi022 ceph-mon[66341]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.116:0/3505700020"}]': finished 2024-03-24T04:32:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:31 smithi022 ceph-mon[66341]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T04:32:31.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:31.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:31 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:31 smithi116 ceph-mon[53251]: from='mds.? [v2:172.21.15.22:6834/2885461479,v1:172.21.15.22:6835/2885461479]' entity='mds.foofs.smithi022.ekdqmp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.116:0/3505700020"}]': finished 2024-03-24T04:32:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:31 smithi116 ceph-mon[53251]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T04:32:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:31.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:31 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:32 smithi116 ceph-mon[53251]: pgmap v346: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T04:32:32.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:32 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:32.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:32 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:32 smithi022 ceph-mon[66341]: pgmap v346: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T04:32:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:32 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:32.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:32 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: pgmap v347: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:33.878 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: pgmap v347: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:34.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:36 smithi116 ceph-mon[53251]: pgmap v348: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T04:32:36.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:36 smithi022 ceph-mon[66341]: pgmap v348: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T04:32:37.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:37 smithi022 ceph-mon[66341]: pgmap v349: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T04:32:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:37 smithi116 ceph-mon[53251]: pgmap v349: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-24T04:32:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi022.iycgyn"}]: dispatch 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi022.iycgyn"}]': finished 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi116.wxmbpt"}]: dispatch 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi116.wxmbpt"}]': finished 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi022.iycgyn"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi022.iycgyn"}]': finished 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi116.wxmbpt"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi116.wxmbpt"}]': finished 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:39.482 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:40 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all nfs 2024-03-24T04:32:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:40 smithi116 ceph-mon[53251]: Upgrade: Setting container_image for all nvmeof 2024-03-24T04:32:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:40 smithi116 ceph-mon[53251]: pgmap v350: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T04:32:40.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:40 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all nfs 2024-03-24T04:32:40.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:40 smithi022 ceph-mon[66341]: Upgrade: Setting container_image for all nvmeof 2024-03-24T04:32:40.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:40 smithi022 ceph-mon[66341]: pgmap v350: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 409 B/s wr, 0 op/s 2024-03-24T04:32:42.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:42 smithi022 ceph-mon[66341]: Upgrade: Updating node-exporter.smithi022 (1/2) 2024-03-24T04:32:42.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:42 smithi022 ceph-mon[66341]: Deploying daemon node-exporter.smithi022 on smithi022 2024-03-24T04:32:42.287 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:42 smithi022 ceph-mon[66341]: pgmap v351: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 278 B/s wr, 0 op/s 2024-03-24T04:32:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:42 smithi116 ceph-mon[53251]: Upgrade: Updating node-exporter.smithi022 (1/2) 2024-03-24T04:32:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:42 smithi116 ceph-mon[53251]: Deploying daemon node-exporter.smithi022 on smithi022 2024-03-24T04:32:42.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:42 smithi116 ceph-mon[53251]: pgmap v351: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 278 B/s wr, 0 op/s 2024-03-24T04:32:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:44 smithi116 ceph-mon[53251]: pgmap v352: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T04:32:44.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:44 smithi022 ceph-mon[66341]: pgmap v352: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T04:32:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:46 smithi116 ceph-mon[53251]: pgmap v353: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T04:32:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:46.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:46 smithi022 ceph-mon[66341]: pgmap v353: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-24T04:32:46.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:46.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:48.428 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:48 smithi116 ceph-mon[53251]: pgmap v354: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:48.428 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:48 smithi116 ceph-mon[53251]: Upgrade: Updating node-exporter.smithi116 (2/2) 2024-03-24T04:32:48.428 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:48 smithi116 ceph-mon[53251]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-24T04:32:48.428 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:48 smithi022 ceph-mon[66341]: pgmap v354: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:48 smithi022 ceph-mon[66341]: Upgrade: Updating node-exporter.smithi116 (2/2) 2024-03-24T04:32:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:48 smithi022 ceph-mon[66341]: Deploying daemon node-exporter.smithi116 on smithi116 2024-03-24T04:32:48.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:32:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:50 smithi116 ceph-mon[53251]: pgmap v355: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:50.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:50 smithi022 ceph-mon[66341]: pgmap v355: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:52 smithi116 ceph-mon[53251]: pgmap v356: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:52.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:52.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:52.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:52 smithi022 ceph-mon[66341]: pgmap v356: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:52.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:32:52.687 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (10m) 21s ago 19m 24.5M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 21s ago 19m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (6m) 24s ago 18m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (10m) 21s ago 19m 54.7M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (71s) 21s ago 13m 33.0M - 19.0.0-1667-gdb0330b1 4f325f8440e9 1d0af2aadefd 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (47s) 24s ago 13m 14.8M - 19.0.0-1667-gdb0330b1 4f325f8440e9 9d19bfdc5f25 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (9m) 21s ago 20m 513M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:32:53.387 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (8m) 24s ago 18m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (8m) 21s ago 21m 59.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (8m) 24s ago 18m 46.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (33s) 21s ago 13m 59.6M - 5.7 4f325f8440e9 8bea989d9586 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (27s) 24s ago 13m 17.2M - 5.7 4f325f8440e9 95fd8e8ab9c2 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 starting - - - - 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 starting - - - - 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (6m) 21s ago 18m 444M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (5m) 21s ago 17m 473M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 21s ago 17m 374M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (4m) 21s ago 16m 410M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (3m) 24s ago 16m 330M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:32:53.388 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (2m) 24s ago 16m 326M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:32:53.389 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (2m) 24s ago 15m 382M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:32:53.389 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (100s) 24s ago 15m 282M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:32:53.389 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (8m) 21s ago 18m 70.9M - 2.33.4 514e6a882f6e 0d5009025ed8 2024-03-24T04:32:53.820 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:32:53.820 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:32:53.820 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:32:53.821 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:32:54.173 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:32:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:32:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:32:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:32:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "nfs", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "mds", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "osd" 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "18/23 daemons upgraded", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:32:54.174 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:32:54.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:54 smithi116 ceph-mon[53251]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:54.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:54 smithi116 ceph-mon[53251]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:54.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:54 smithi116 ceph-mon[53251]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:54.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:54 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/4093616172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:54 smithi022 ceph-mon[66341]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:54 smithi022 ceph-mon[66341]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:54 smithi022 ceph-mon[66341]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:54.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:54 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/4093616172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:32:54.599 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:32:55.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:55 smithi022 ceph-mon[66341]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:55.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:55 smithi022 ceph-mon[66341]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:55.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:55 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2405839669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:32:55.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:55.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:55 smithi116 ceph-mon[53251]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:55 smithi116 ceph-mon[53251]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:32:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:55 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2405839669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:32:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:55.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:56 smithi116 ceph-mon[53251]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:32:56.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:56 smithi022 ceph-mon[66341]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:32:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:57.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:57.518 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:57.518 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:58 smithi116 ceph-mon[53251]: pgmap v359: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:58.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:32:58 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:58 smithi022 ceph-mon[66341]: pgmap v359: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:32:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:32:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:32:58 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:00 smithi022 ceph-mon[66341]: pgmap v360: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:00.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:00 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:00 smithi116 ceph-mon[53251]: pgmap v360: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:00.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:00.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:00 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:01.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:01 smithi022 ceph-mon[66341]: pgmap v361: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:01 smithi116 ceph-mon[53251]: pgmap v361: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:04.216 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.216 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:04.216 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.216 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: pgmap v362: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.217 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:04 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: pgmap v362: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:04.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:04 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:06.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:06 smithi022 ceph-mon[66341]: Upgrade: Updating prometheus.smithi022 2024-03-24T04:33:06.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:06 smithi022 ceph-mon[66341]: pgmap v363: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:06 smithi116 ceph-mon[53251]: Upgrade: Updating prometheus.smithi022 2024-03-24T04:33:06.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:06 smithi116 ceph-mon[53251]: pgmap v363: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:07.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:07 smithi022 ceph-mon[66341]: Deploying daemon prometheus.smithi022 on smithi022 2024-03-24T04:33:07.403 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:07 smithi022 ceph-mon[66341]: pgmap v364: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:07 smithi116 ceph-mon[53251]: Deploying daemon prometheus.smithi022 on smithi022 2024-03-24T04:33:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:07 smithi116 ceph-mon[53251]: pgmap v364: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:10.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:10 smithi022 ceph-mon[66341]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:10 smithi116 ceph-mon[53251]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:12.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:11 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:12.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:12.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:12.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:11 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:13.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:13 smithi022 ceph-mon[66341]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:13 smithi116 ceph-mon[53251]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:15 smithi022 ceph-mon[66341]: pgmap v368: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:15 smithi116 ceph-mon[53251]: pgmap v368: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:17.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:17.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:17 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:17.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:17 smithi116 ceph-mon[53251]: pgmap v369: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:18.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:18.047 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:17 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:18.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:17 smithi022 ceph-mon[66341]: pgmap v369: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:19.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:19.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:19.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:19.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:19.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:19.224 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:20.048 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:19 smithi022 ceph-mon[66341]: pgmap v370: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:20.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:19 smithi116 ceph-mon[53251]: pgmap v370: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:21.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:21 smithi022 ceph-mon[66341]: pgmap v371: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:21 smithi116 ceph-mon[53251]: pgmap v371: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:22.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.484 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.485 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:22 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:22.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:22.725 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:22 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:23.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:23 smithi022 ceph-mon[66341]: Upgrade: Updating alertmanager.smithi022 2024-03-24T04:33:23.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:23 smithi022 ceph-mon[66341]: Deploying daemon alertmanager.smithi022 on smithi022 2024-03-24T04:33:23.690 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:23 smithi022 ceph-mon[66341]: pgmap v372: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:23 smithi116 ceph-mon[53251]: Upgrade: Updating alertmanager.smithi022 2024-03-24T04:33:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:23 smithi116 ceph-mon[53251]: Deploying daemon alertmanager.smithi022 on smithi022 2024-03-24T04:33:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:23 smithi116 ceph-mon[53251]: pgmap v372: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:24.964 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:33:25.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:25 smithi022 ceph-mon[66341]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:25.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:25 smithi022 ceph-mon[66341]: pgmap v373: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:25.642 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:33:25.642 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (11m) 9s ago 20m 24.6M - 0.23.0 ba2b418f427c 997946ad3587 2024-03-24T04:33:25.642 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 9s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (7m) 30s ago 19m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (11m) 9s ago 19m 55.5M - 8.3.5 dad864ee21e9 5592d6d2dcb3 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (103s) 9s ago 14m 33.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 1d0af2aadefd 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (79s) 30s ago 14m 14.9M - 19.0.0-1667-gdb0330b1 4f325f8440e9 9d19bfdc5f25 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (10m) 9s ago 21m 517M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (9m) 30s ago 19m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (9m) 9s ago 21m 60.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (8m) 30s ago 19m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (65s) 9s ago 13m 60.2M - 5.7 4f325f8440e9 8bea989d9586 2024-03-24T04:33:25.643 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (60s) 30s ago 13m 61.6M - 5.7 4f325f8440e9 95fd8e8ab9c2 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (40s) 9s ago 19m 8518k - 1.5.0 0da6a335fe13 9645b05b81f9 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (34s) 30s ago 19m 3577k - 1.5.0 0da6a335fe13 dd6401de2285 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (6m) 9s ago 18m 446M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (6m) 9s ago 18m 473M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (5m) 9s ago 17m 376M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (4m) 9s ago 17m 411M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (4m) 30s ago 16m 331M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (3m) 30s ago 16m 327M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (2m) 30s ago 16m 382M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (2m) 30s ago 15m 282M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:33:25.644 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (15s) 9s ago 19m 38.7M - 2.43.0 a07b618ecd1d 5a06e7ef4b20 2024-03-24T04:33:25.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:25 smithi116 ceph-mon[53251]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:25.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:25 smithi116 ceph-mon[53251]: pgmap v373: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:26.052 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:33:26.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:33:26.054 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [ 2024-03-24T04:33:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "nfs", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "mds", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "mon", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "osd" 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "18/23 daemons upgraded", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:33:26.394 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:33:26.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:26 smithi022 ceph-mon[66341]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:26.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:26 smithi022 ceph-mon[66341]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:26.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:26 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/4245327604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:26 smithi116 ceph-mon[53251]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:26 smithi116 ceph-mon[53251]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:26.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:26 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/4245327604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:26.809 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:33:27.468 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:27 smithi022 ceph-mon[66341]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:27.468 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:27 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3581258348' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:33:27.468 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:27 smithi022 ceph-mon[66341]: pgmap v374: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:27 smithi116 ceph-mon[53251]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:27 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3581258348' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:33:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:27 smithi116 ceph-mon[53251]: pgmap v374: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:30 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:30.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:30 smithi022 ceph-mon[66341]: pgmap v375: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:30.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:30 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:30.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:30 smithi116 ceph-mon[53251]: pgmap v375: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:31.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:31 smithi022 ceph-mon[66341]: pgmap v376: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:31 smithi116 ceph-mon[53251]: pgmap v376: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:34 smithi116 ceph-mon[53251]: pgmap v377: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:34.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:34 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:34 smithi022 ceph-mon[66341]: pgmap v377: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:34.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:34 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:36.444 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:36 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:36.444 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:36 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:36.445 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:36 smithi022 ceph-mon[66341]: pgmap v378: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:36 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:36 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:36 smithi116 ceph-mon[53251]: pgmap v378: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:37.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:37 smithi022 ceph-mon[66341]: pgmap v379: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:37.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:37 smithi116 ceph-mon[53251]: pgmap v379: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.725 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.726 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.727 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:39 smithi022 ceph-mon[66341]: pgmap v380: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:39.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T04:33:39.974 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.975 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:39.976 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:39 smithi116 ceph-mon[53251]: pgmap v380: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:41.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:41 smithi022 ceph-mon[66341]: Upgrade: Updating grafana.smithi022 2024-03-24T04:33:41.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:41 smithi022 ceph-mon[66341]: Deploying daemon grafana.smithi022 on smithi022 2024-03-24T04:33:41.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:41 smithi022 ceph-mon[66341]: pgmap v381: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:41.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:41 smithi116 ceph-mon[53251]: Upgrade: Updating grafana.smithi022 2024-03-24T04:33:41.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:41 smithi116 ceph-mon[53251]: Deploying daemon grafana.smithi022 on smithi022 2024-03-24T04:33:41.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:41 smithi116 ceph-mon[53251]: pgmap v381: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:43.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:43 smithi022 ceph-mon[66341]: pgmap v382: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:43.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:43 smithi116 ceph-mon[53251]: pgmap v382: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:46 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:46.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:46 smithi022 ceph-mon[66341]: pgmap v383: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:46 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:46.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:46 smithi116 ceph-mon[53251]: pgmap v383: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:47.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:47 smithi022 ceph-mon[66341]: pgmap v384: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:47.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:47 smithi116 ceph-mon[53251]: pgmap v384: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:48.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:33:49.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:49 smithi022 ceph-mon[66341]: pgmap v385: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:49.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:49 smithi116 ceph-mon[53251]: pgmap v385: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:51 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:51.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:51 smithi116 ceph-mon[53251]: pgmap v386: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:51.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:51.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:51 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:51.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:51 smithi022 ceph-mon[66341]: pgmap v386: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:52.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:52.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:52 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:52.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:52 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:54.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:53 smithi022 ceph-mon[66341]: pgmap v387: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:54.223 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:53 smithi116 ceph-mon[53251]: pgmap v387: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:33:56.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.299 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: Upgrade: Finalizing container_image settings 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-24T04:33:56.300 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: pgmap v388: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-24T04:33:56.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: Upgrade: Complete! 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.302 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.303 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:55 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.474 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: Upgrade: Finalizing container_image settings 2024-03-24T04:33:56.475 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: pgmap v388: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-24T04:33:56.476 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.477 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: Upgrade: Complete! 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:33:56.478 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:33:56.479 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:33:56.479 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:56.479 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:55 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:57.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:57 smithi022 ceph-mon[66341]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:57.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:57 smithi022 ceph-mon[66341]: pgmap v389: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:57 smithi116 ceph-mon[53251]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:33:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:57 smithi116 ceph-mon[53251]: pgmap v389: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:57.864 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T04:33:59.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:59 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:59.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:33:59 smithi116 ceph-mon[53251]: pgmap v390: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:59.791 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:59 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:33:59.791 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:33:59 smithi022 ceph-mon[66341]: pgmap v390: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:33:59.791 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (31s) 9s ago 20m 17.6M - 0.25.0 c8568f914cd2 64e966f5e968 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (8m) 9s ago 20m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (8m) 64s ago 20m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (15s) 9s ago 20m 53.9M - 9.4.12 a0ea5e3c46d9 7ed9036e21f9 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (2m) 9s ago 14m 33.6M - 19.0.0-1667-gdb0330b1 4f325f8440e9 1d0af2aadefd 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (113s) 64s ago 14m 14.9M - 19.0.0-1667-gdb0330b1 4f325f8440e9 9d19bfdc5f25 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (10m) 9s ago 22m 517M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (9m) 64s ago 20m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (9m) 9s ago 22m 61.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (9m) 64s ago 19m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (99s) 9s ago 14m 60.4M - 5.7 4f325f8440e9 8bea989d9586 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (94s) 64s ago 14m 61.6M - 5.7 4f325f8440e9 95fd8e8ab9c2 2024-03-24T04:33:59.792 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (74s) 9s ago 20m 8476k - 1.5.0 0da6a335fe13 9645b05b81f9 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (68s) 64s ago 19m 3577k - 1.5.0 0da6a335fe13 dd6401de2285 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (7m) 9s ago 19m 447M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (6m) 9s ago 18m 473M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (5m) 9s ago 18m 377M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (5m) 9s ago 17m 412M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (4m) 64s ago 17m 331M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (3m) 64s ago 17m 327M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (3m) 64s ago 16m 382M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (2m) 64s ago 16m 282M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:33:59.793 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (49s) 9s ago 19m 48.9M - 2.43.0 a07b618ecd1d 5a06e7ef4b20 2024-03-24T04:34:00.367 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T04:34:00.679 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:00 smithi022 ceph-mon[66341]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:34:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:00 smithi116 ceph-mon[53251]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:34:01.688 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:01 smithi022 ceph-mon[66341]: pgmap v391: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:01.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:01 smithi116 ceph-mon[53251]: pgmap v391: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:02.413 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:34:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:34:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:34:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:34:02.414 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:34:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:02 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3520953340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:34:02.740 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:02 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3520953340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:34:03.067 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-24T04:34:03.631 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:03.631 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:03 smithi022 ceph-mon[66341]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:03.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:03 smithi116 ceph-mon[53251]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:04.727 INFO:teuthology.orchestra.run.smithi022.stdout:wait for servicemap items w/ changing names to refresh 2024-03-24T04:34:05.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:05 smithi022 ceph-mon[66341]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:05.400 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'sleep 60' 2024-03-24T04:34:05.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:05 smithi116 ceph-mon[53251]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:07.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:07 smithi022 ceph-mon[66341]: pgmap v394: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:07.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:07 smithi116 ceph-mon[53251]: pgmap v394: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:09.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:09 smithi022 ceph-mon[66341]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:09.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:09 smithi116 ceph-mon[53251]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:11.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:11 smithi022 ceph-mon[66341]: pgmap v396: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:11.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:11 smithi116 ceph-mon[53251]: pgmap v396: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:13.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:13 smithi022 ceph-mon[66341]: pgmap v397: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:13 smithi116 ceph-mon[53251]: pgmap v397: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:15.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:15 smithi022 ceph-mon[66341]: pgmap v398: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:15.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:15 smithi116 ceph-mon[53251]: pgmap v398: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:17.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:17 smithi022 ceph-mon[66341]: pgmap v399: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:17.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:17 smithi116 ceph-mon[53251]: pgmap v399: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:18.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:18.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:19.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:19 smithi022 ceph-mon[66341]: pgmap v400: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:19 smithi116 ceph-mon[53251]: pgmap v400: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:21.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:21 smithi022 ceph-mon[66341]: pgmap v401: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:21 smithi116 ceph-mon[53251]: pgmap v401: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:23.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:23 smithi022 ceph-mon[66341]: pgmap v402: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:23.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:23 smithi116 ceph-mon[53251]: pgmap v402: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:25.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:25 smithi022 ceph-mon[66341]: pgmap v403: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:25.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:25 smithi116 ceph-mon[53251]: pgmap v403: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:27.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:27 smithi022 ceph-mon[66341]: pgmap v404: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:27.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:27 smithi116 ceph-mon[53251]: pgmap v404: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:29.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:29 smithi022 ceph-mon[66341]: pgmap v405: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:29 smithi116 ceph-mon[53251]: pgmap v405: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:31.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:31 smithi022 ceph-mon[66341]: pgmap v406: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:31 smithi116 ceph-mon[53251]: pgmap v406: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:33.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:33.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:34.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:34 smithi116 ceph-mon[53251]: pgmap v407: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:34.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:34 smithi022 ceph-mon[66341]: pgmap v407: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:36 smithi116 ceph-mon[53251]: pgmap v408: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:36.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:36 smithi022 ceph-mon[66341]: pgmap v408: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:38 smithi116 ceph-mon[53251]: pgmap v409: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:38.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:38 smithi022 ceph-mon[66341]: pgmap v409: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:40.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:40 smithi116 ceph-mon[53251]: pgmap v410: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:40.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:40 smithi022 ceph-mon[66341]: pgmap v410: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:42.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:42 smithi116 ceph-mon[53251]: pgmap v411: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:42.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:42 smithi022 ceph-mon[66341]: pgmap v411: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:44.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:44 smithi116 ceph-mon[53251]: pgmap v412: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:44.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:44 smithi022 ceph-mon[66341]: pgmap v412: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:46.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:46 smithi116 ceph-mon[53251]: pgmap v413: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:46.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:46 smithi022 ceph-mon[66341]: pgmap v413: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:48 smithi116 ceph-mon[53251]: pgmap v414: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:48.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:48 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:48 smithi022 ceph-mon[66341]: pgmap v414: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:48.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:48 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:34:50.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:50 smithi116 ceph-mon[53251]: pgmap v415: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:50.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:50 smithi022 ceph-mon[66341]: pgmap v415: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:52.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:52 smithi116 ceph-mon[53251]: pgmap v416: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:52.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:52 smithi022 ceph-mon[66341]: pgmap v416: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:54.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:54 smithi022 ceph-mon[66341]: pgmap v417: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:54.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:54 smithi116 ceph-mon[53251]: pgmap v417: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:56 smithi022 ceph-mon[66341]: pgmap v418: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:34:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:56.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:56 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:56 smithi116 ceph-mon[53251]: pgmap v418: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:34:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T04:34:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:56.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:56 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:57.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:34:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:34:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:57.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:57 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T04:34:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T04:34:57.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:57.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:57 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' 2024-03-24T04:34:58.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:34:58 smithi022 ceph-mon[66341]: pgmap v419: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:34:58.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:34:58 smithi116 ceph-mon[53251]: pgmap v419: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:00.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:00 smithi022 ceph-mon[66341]: pgmap v420: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:00.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:00 smithi116 ceph-mon[53251]: pgmap v420: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:02.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:02 smithi022 ceph-mon[66341]: pgmap v421: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:02.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:02 smithi116 ceph-mon[53251]: pgmap v421: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:03.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:03 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:03.624 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:03 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:04.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:04 smithi116 ceph-mon[53251]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:04.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:04 smithi022 ceph-mon[66341]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:05.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:05 smithi022 ceph-mon[66341]: pgmap v423: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:05.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:05 smithi116 ceph-mon[53251]: pgmap v423: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:07.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:07 smithi022 ceph-mon[66341]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:07.635 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T04:35:07.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:07 smithi116 ceph-mon[53251]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (101s) 79s ago 21m 17.6M - 0.25.0 c8568f914cd2 64e966f5e968 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (9m) 79s ago 21m 6660k - 19.0.0-1667-gdb0330b1 4f325f8440e9 62c643077dec 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi116 smithi116 running (9m) 2m ago 21m 6665k - 19.0.0-1667-gdb0330b1 4f325f8440e9 ec82aa5b5625 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (84s) 79s ago 21m 53.9M - 9.4.12 a0ea5e3c46d9 7ed9036e21f9 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi022.ekdqmp smithi022 running (3m) 79s ago 16m 33.6M - 19.0.0-1667-gdb0330b1 4f325f8440e9 1d0af2aadefd 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:mds.foofs.smithi116.kvgpsf smithi116 running (3m) 2m ago 15m 14.9M - 19.0.0-1667-gdb0330b1 4f325f8440e9 9d19bfdc5f25 2024-03-24T04:35:09.493 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.ijpjvk smithi022 *:8443,9283,8765 running (12m) 79s ago 23m 517M - 19.0.0-1667-gdb0330b1 4f325f8440e9 a19e585dceac 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi116.oxktjt smithi116 *:8443,9283,8765 running (11m) 2m ago 21m 453M - 19.0.0-1667-gdb0330b1 4f325f8440e9 466aa7d07d27 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (10m) 79s ago 23m 61.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 ec3e67d1d231 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi116 smithi116 running (10m) 2m ago 21m 47.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 1d02fa4fc4ec 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.0.0.smithi022.iycgyn smithi022 *:2049 running (2m) 79s ago 15m 60.4M - 5.7 4f325f8440e9 8bea989d9586 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:nfs.foo.1.0.smithi116.wxmbpt smithi116 *:2049 running (2m) 2m ago 15m 61.6M - 5.7 4f325f8440e9 95fd8e8ab9c2 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (2m) 79s ago 21m 8476k - 1.5.0 0da6a335fe13 9645b05b81f9 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi116 smithi116 *:9100 running (2m) 2m ago 20m 3577k - 1.5.0 0da6a335fe13 dd6401de2285 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (8m) 79s ago 20m 447M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0596bedc58f1 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (7m) 79s ago 19m 473M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 41cef8e3a807 2024-03-24T04:35:09.494 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (7m) 79s ago 19m 377M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 1baf3f8fdbc2 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (6m) 79s ago 19m 412M 1937M 19.0.0-1667-gdb0330b1 4f325f8440e9 0a452e1436b9 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi116 running (5m) 2m ago 18m 331M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 97425f93c865 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi116 running (5m) 2m ago 18m 327M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 bb9dd19044c4 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi116 running (4m) 2m ago 17m 382M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 7d40be1a1ff8 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi116 running (3m) 2m ago 17m 282M 2704M 19.0.0-1667-gdb0330b1 4f325f8440e9 a8a47b2d03e5 2024-03-24T04:35:09.495 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (119s) 79s ago 20m 48.9M - 2.43.0 a07b618ecd1d 5a06e7ef4b20 2024-03-24T04:35:09.495 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:09 smithi022 ceph-mon[66341]: pgmap v425: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:09.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:09 smithi116 ceph-mon[53251]: pgmap v425: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:10.104 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade status' 2024-03-24T04:35:10.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:10 smithi022 ceph-mon[66341]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:10.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:10 smithi116 ceph-mon[53251]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:11.572 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:11 smithi022 ceph-mon[66341]: pgmap v426: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:11.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:11 smithi116 ceph-mon[53251]: pgmap v426: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:12.103 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:35:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": null, 2024-03-24T04:35:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": false, 2024-03-24T04:35:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "", 2024-03-24T04:35:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T04:35:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": null, 2024-03-24T04:35:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "", 2024-03-24T04:35:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T04:35:12.104 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:35:12.433 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:12 smithi022 ceph-mon[66341]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:12.669 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph health detail' 2024-03-24T04:35:12.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:12 smithi116 ceph-mon[53251]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:13.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:13 smithi116 ceph-mon[53251]: pgmap v427: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:13.793 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:13 smithi022 ceph-mon[66341]: pgmap v427: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:14.613 INFO:teuthology.orchestra.run.smithi022.stdout:HEALTH_OK 2024-03-24T04:35:14.971 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:14 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1760513470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:35:14.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:14 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1760513470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T04:35:15.236 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T04:35:15.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:15 smithi022 ceph-mon[66341]: pgmap v428: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:15.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:15 smithi116 ceph-mon[53251]: pgmap v428: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T04:35:17.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 14 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:35:17.136 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T04:35:17.434 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:17 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/1450811495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:17.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:17 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/1450811495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:17.733 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-24T04:35:18.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:18 smithi022 ceph-mon[66341]: pgmap v429: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:18.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:18 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:18 smithi116 ceph-mon[53251]: pgmap v429: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:18.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:18 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:19.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:19 smithi022 ceph-mon[66341]: pgmap v430: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:19.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:19 smithi116 ceph-mon[53251]: pgmap v430: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:19.952 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T04:35:20.525 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:20 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2056004346' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:20.548 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-24T04:35:20.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:20 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2056004346' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:21.416 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:21 smithi022 ceph-mon[66341]: pgmap v431: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:21.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:21 smithi116 ceph-mon[53251]: pgmap v431: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:22.508 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)" 2024-03-24T04:35:22.797 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:22 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/2803469311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:22.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:22 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/2803469311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T04:35:23.106 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-03-24T04:35:23.107 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-03-24T04:35:23.108 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- ceph orch ls -f json 2024-03-24T04:35:23.734 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:23 smithi022 ceph-mon[66341]: pgmap v432: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:23.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:23 smithi116 ceph-mon[53251]: pgmap v432: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:25.061 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T04:35:25.061 INFO:teuthology.orchestra.run.smithi022.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-24T04:12:37.186370Z", "last_refresh": "2024-03-24T04:33:50.406669Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-24T04:12:32.129867Z", "last_refresh": "2024-03-24T04:32:55.164400Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-03-24T04:12:34.573641Z", "last_refresh": "2024-03-24T04:33:50.406850Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-24T04:19:02.287150Z", "last_refresh": "2024-03-24T04:32:55.165066Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-24T04:12:30.060499Z", "last_refresh": "2024-03-24T04:32:55.164504Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi022:172.21.15.22=smithi022", "smithi116:172.21.15.116=smithi116"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-24T04:13:20.654303Z", "last_refresh": "2024-03-24T04:32:55.164586Z", "running": 2, "size": 2}}, {"events": ["2024-03-24T04:34:56.602709Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-03-24T04:19:22.380802Z", "last_refresh": "2024-03-24T04:32:55.165144Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-24T04:12:35.982152Z", "last_refresh": "2024-03-24T04:32:55.164666Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", "last_refresh": "2024-03-24T04:32:55.164745Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-24T04:12:33.276375Z", "last_refresh": "2024-03-24T04:33:50.407020Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-24T04:35:25.446 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:25 smithi022 ceph-mon[66341]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:35:25.446 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:25 smithi022 ceph-mon[66341]: pgmap v433: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:25.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:25 smithi116 ceph-mon[53251]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T04:35:25.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:25 smithi116 ceph-mon[53251]: pgmap v433: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T04:35:25.742 INFO:tasks.cephadm:nfs.foo has 2/2 2024-03-24T04:35:25.743 DEBUG:teuthology.parallel:result is None 2024-03-24T04:35:25.743 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-24T04:35:25.754 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:35:25.755 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-03-24T04:35:25.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ umount /mnt/foo 2024-03-24T04:35:25.797 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T04:35:25.807 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:35:25.808 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-03-24T04:35:27.358 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:27 smithi022 ceph-mon[66341]: pgmap v434: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:27.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:27 smithi116 ceph-mon[53251]: pgmap v434: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:27.820 INFO:teuthology.orchestra.run.smithi022.stdout: "foo" 2024-03-24T04:35:28.402 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:28 smithi022 ceph-mon[66341]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:28.426 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-03-24T04:35:28.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:28 smithi116 ceph-mon[53251]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:29.693 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:29 smithi022 ceph-mon[66341]: pgmap v435: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:29.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:29 smithi116 ceph-mon[53251]: pgmap v435: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:30.470 INFO:teuthology.orchestra.run.smithi022.stdout:[ 2024-03-24T04:35:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T04:35:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "access_type": "RW", 2024-03-24T04:35:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "clients": [], 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_id": "foo", 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "export_id": 1, 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "fsal": { 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "foofs", 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "CEPH", 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "user_id": "nfs.foo.1" 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "protocols": [ 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: 4 2024-03-24T04:35:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "pseudo": "/fake", 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "security_label": true, 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "squash": "none", 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "transports": [ 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "TCP" 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T04:35:30.472 INFO:teuthology.orchestra.run.smithi022.stdout:] 2024-03-24T04:35:31.126 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-03-24T04:35:31.438 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:31 smithi022 ceph-mon[66341]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:31.438 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:31 smithi022 ceph-mon[66341]: pgmap v436: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:31 smithi116 ceph-mon[53251]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T04:35:31.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:31 smithi116 ceph-mon[53251]: pgmap v436: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-24T04:35:32.828 INFO:teuthology.orchestra.run.smithi022.stdout:foo rec-0000000000000004:nfs.foo.1 2024-03-24T04:35:32.828 INFO:teuthology.orchestra.run.smithi022.stdout:foo grace 2024-03-24T04:35:32.828 INFO:teuthology.orchestra.run.smithi022.stdout:foo rec-0000000000000004:nfs.foo.0 2024-03-24T04:35:32.828 INFO:teuthology.orchestra.run.smithi022.stdout:foo conf-nfs.foo 2024-03-24T04:35:32.828 INFO:teuthology.orchestra.run.smithi022.stdout:foo export-1 2024-03-24T04:35:33.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:33 smithi022 ceph-mon[66341]: mgrmap e44: smithi022.ijpjvk(active, since 10m), standbys: smithi116.oxktjt 2024-03-24T04:35:33.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:33 smithi022 ceph-mon[66341]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:33.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:33 smithi022 ceph-mon[66341]: pgmap v437: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T04:35:33.446 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T04:35:33.457 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:35:33.457 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 -- bash -c 'set -ex 2024-03-24T04:35:33.458 DEBUG:teuthology.orchestra.run.smithi022:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-03-24T04:35:33.458 DEBUG:teuthology.orchestra.run.smithi022:> ' 2024-03-24T04:35:33.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:33 smithi116 ceph-mon[53251]: mgrmap e44: smithi022.ijpjvk(active, since 10m), standbys: smithi116.oxktjt 2024-03-24T04:35:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:33 smithi116 ceph-mon[53251]: from='mgr.34102 172.21.15.22:0/3702053615' entity='mgr.smithi022.ijpjvk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T04:35:33.724 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:33 smithi116 ceph-mon[53251]: pgmap v437: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T04:35:35.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-03-24T04:35:35.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ [[ 6 -gt 2 ]] 2024-03-24T04:35:35.547 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:35 smithi022 ceph-mon[66341]: pgmap v438: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-24T04:35:35.723 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:35 smithi116 ceph-mon[53251]: pgmap v438: 97 pgs: 97 active+clean; 227 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-24T04:35:35.997 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T04:35:36.008 INFO:tasks.cephadm:Teardown begin 2024-03-24T04:35:36.009 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:35:36.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:35:36.074 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T04:35:36.075 DEBUG:teuthology.orchestra.run.smithi022:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T04:35:36.094 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T04:35:36.131 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T04:35:36.132 INFO:tasks.cephadm.mon.smithi022:Stopping mon.smithi022... 2024-03-24T04:35:36.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022 2024-03-24T04:35:36.297 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:36 smithi022 systemd[1]: Stopping Ceph mon.smithi022 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:35:36.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:36 smithi116 ceph-mon[53251]: from='client.? 172.21.15.22:0/3622066443' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-24T04:35:36.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:36 smithi022 ceph-mon[66341]: from='client.? 172.21.15.22:0/3622066443' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-24T04:35:36.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:36 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[66334]: 2024-03-24T04:35:36.426+0000 7f7da1d24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi022 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:35:36.548 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:36 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022[66334]: 2024-03-24T04:35:36.426+0000 7f7da1d24700 -1 mon.smithi022@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T04:35:36.961 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:36 smithi022 podman[111450]: 2024-03-24 04:35:36.68162908 +0000 UTC m=+0.361207949 container died ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-24T04:35:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:37 smithi022 podman[111450]: 2024-03-24 04:35:37.043059136 +0000 UTC m=+0.722638004 container cleanup ec3e67d1d2315987bc47dc5c2df418a0156b3303ae8ab0c222c0aa8472f9cd3e (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T04:35:37.298 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 04:35:37 smithi022 bash[111450]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi022 2024-03-24T04:35:37.528 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi022.service' 2024-03-24T04:35:37.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:35:37.574 INFO:tasks.cephadm.mon.smithi022:Stopped mon.smithi022 2024-03-24T04:35:37.574 INFO:tasks.cephadm.mon.smithi116:Stopping mon.smithi116... 2024-03-24T04:35:37.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi116 2024-03-24T04:35:37.964 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:37 smithi116 systemd[1]: Stopping Ceph mon.smithi116 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:35:38.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116[53247]: 2024-03-24T04:35:37.963+0000 7f6f458cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi116 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:35:38.219 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116[53247]: 2024-03-24T04:35:37.963+0000 7f6f458cf700 -1 mon.smithi116@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T04:35:38.473 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:38 smithi116 podman[78707]: 2024-03-24 04:35:38.219508503 +0000 UTC m=+0.447256487 container died 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T04:35:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:38 smithi116 podman[78707]: 2024-03-24 04:35:38.655449363 +0000 UTC m=+0.883197399 container cleanup 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T04:35:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:38 smithi116 bash[78707]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116 2024-03-24T04:35:38.973 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Mar 24 04:35:38 smithi116 podman[78721]: 2024-03-24 04:35:38.865164922 +0000 UTC m=+0.645953915 container remove 1d02fa4fc4ecaf1f04a1c0038dfdeee61e246006b202074f7d11fa2edf860b22 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-mon-smithi116, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T04:35:39.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@mon.smithi116.service' 2024-03-24T04:35:39.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:35:39.132 INFO:tasks.cephadm.mon.smithi116:Stopped mon.smithi116 2024-03-24T04:35:39.132 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T04:35:39.133 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.0 2024-03-24T04:35:39.422 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:39 smithi022 systemd[1]: Stopping Ceph osd.0 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:35:39.797 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:39 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:35:39.421+0000 7f3e862ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:35:39.798 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:39 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:35:39.421+0000 7f3e862ce700 -1 osd.0 176 *** Got signal Terminated *** 2024-03-24T04:35:39.798 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:39 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0[72446]: 2024-03-24T04:35:39.421+0000 7f3e862ce700 -1 osd.0 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:35:45.215 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:44 smithi022 podman[111582]: 2024-03-24 04:35:44.929071552 +0000 UTC m=+5.602339556 container died 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T04:35:45.547 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:45 smithi022 podman[111582]: 2024-03-24 04:35:45.313990422 +0000 UTC m=+5.987258420 container cleanup 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:35:45.548 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:45 smithi022 bash[111582]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0 2024-03-24T04:35:46.047 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:45 smithi022 podman[111597]: 2024-03-24 04:35:45.649077754 +0000 UTC m=+0.723062151 container remove 0596bedc58f15be7d319f113cf10ccf7a019c5b171776f3d7ed9796699467968 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0, ceph=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T04:35:46.460 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111692]: 2024-03-24 04:35:45.989977441 +0000 UTC m=+0.018643259 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:35:46.461 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111692]: 2024-03-24 04:35:46.091780655 +0000 UTC m=+0.120446487 container create acc99cac77e266ccd1a4b52a8e9dd2a095b293c9261fae467389269f51dbd0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T04:35:46.461 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111692]: 2024-03-24 04:35:46.461117471 +0000 UTC m=+0.489783271 container init acc99cac77e266ccd1a4b52a8e9dd2a095b293c9261fae467389269f51dbd0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-24T04:35:46.761 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111692]: 2024-03-24 04:35:46.464569697 +0000 UTC m=+0.493235490 container start acc99cac77e266ccd1a4b52a8e9dd2a095b293c9261fae467389269f51dbd0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T04:35:46.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111692]: 2024-03-24 04:35:46.536055357 +0000 UTC m=+0.564721156 container attach acc99cac77e266ccd1a4b52a8e9dd2a095b293c9261fae467389269f51dbd0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T04:35:47.047 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 04:35:46 smithi022 podman[111718]: 2024-03-24 04:35:46.787098993 +0000 UTC m=+0.019408139 container died acc99cac77e266ccd1a4b52a8e9dd2a095b293c9261fae467389269f51dbd0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-24T04:35:47.351 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T04:35:47.397 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:35:47.397 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T04:35:47.397 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T04:35:47.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.1 2024-03-24T04:35:47.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:47 smithi022 systemd[1]: Stopping Ceph osd.1 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:35:48.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:47 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:35:47.759+0000 7f1bd6712700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:35:48.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:47 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:35:47.759+0000 7f1bd6712700 -1 osd.1 176 *** Got signal Terminated *** 2024-03-24T04:35:48.048 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:47 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1[74999]: 2024-03-24T04:35:47.759+0000 7f1bd6712700 -1 osd.1 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:35:53.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:53 smithi022 podman[111830]: 2024-03-24 04:35:53.226450263 +0000 UTC m=+5.601812934 container died 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-db0330b, ceph=True) 2024-03-24T04:35:53.797 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:53 smithi022 podman[111830]: 2024-03-24 04:35:53.59296476 +0000 UTC m=+5.968327444 container cleanup 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T04:35:53.798 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:53 smithi022 bash[111830]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1 2024-03-24T04:35:54.164 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:53 smithi022 podman[111848]: 2024-03-24 04:35:53.843984787 +0000 UTC m=+0.615587128 container remove 41cef8e3a807b13bd5a9cd3afdb99e7cef8e295a377fe9c7f637d7e3e883dc13 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T04:35:54.456 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.067640005 +0000 UTC m=+0.018077794 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:35:54.456 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.186911523 +0000 UTC m=+0.137349295 container create 54dd6d9971efe466a87dc445ad1b138ea0c8ba234580dfe8d4b1f7dc93120760 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T04:35:54.456 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.455788467 +0000 UTC m=+0.406226263 container init 54dd6d9971efe466a87dc445ad1b138ea0c8ba234580dfe8d4b1f7dc93120760 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T04:35:54.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.458868173 +0000 UTC m=+0.409305946 container start 54dd6d9971efe466a87dc445ad1b138ea0c8ba234580dfe8d4b1f7dc93120760 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T04:35:54.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.556416099 +0000 UTC m=+0.506853887 container attach 54dd6d9971efe466a87dc445ad1b138ea0c8ba234580dfe8d4b1f7dc93120760 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:35:55.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 04:35:54 smithi022 podman[111938]: 2024-03-24 04:35:54.75933148 +0000 UTC m=+0.709769257 container died 54dd6d9971efe466a87dc445ad1b138ea0c8ba234580dfe8d4b1f7dc93120760 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:35:55.383 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T04:35:55.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:35:55.427 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T04:35:55.427 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T04:35:55.428 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.2 2024-03-24T04:35:55.771 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:35:55 smithi022 systemd[1]: Stopping Ceph osd.2 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:35:56.047 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:35:55 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:35:55.770+0000 7fd7fde9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:35:56.047 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:35:55 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:35:55.770+0000 7fd7fde9c700 -1 osd.2 176 *** Got signal Terminated *** 2024-03-24T04:35:56.048 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:35:55 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2[77719]: 2024-03-24T04:35:55.770+0000 7fd7fde9c700 -1 osd.2 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:01.547 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:01 smithi022 podman[112042]: 2024-03-24 04:36:01.277840755 +0000 UTC m=+5.629618915 container died 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-24T04:36:01.939 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:01 smithi022 podman[112042]: 2024-03-24 04:36:01.67933556 +0000 UTC m=+6.031113708 container cleanup 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:36:01.939 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:01 smithi022 bash[112042]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2 2024-03-24T04:36:02.296 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:01 smithi022 podman[112059]: 2024-03-24 04:36:01.93942487 +0000 UTC m=+0.657167137 container remove 1baf3f8fdbc2f476fae00e77f1a8e78005ad6efb67f437f466c718b0331445dd (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T04:36:02.547 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.199519603 +0000 UTC m=+0.017834211 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:02.547 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.307001854 +0000 UTC m=+0.125316490 container create 7297ea630096102f6f29cd8d9b972e26c15664644d4c5a834f1ad0dd7467fef4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:36:02.548 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.501555531 +0000 UTC m=+0.319870130 container init 7297ea630096102f6f29cd8d9b972e26c15664644d4c5a834f1ad0dd7467fef4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T04:36:02.548 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.50539602 +0000 UTC m=+0.323710615 container start 7297ea630096102f6f29cd8d9b972e26c15664644d4c5a834f1ad0dd7467fef4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T04:36:03.047 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.584616829 +0000 UTC m=+0.402931445 container attach 7297ea630096102f6f29cd8d9b972e26c15664644d4c5a834f1ad0dd7467fef4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:36:03.047 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 04:36:02 smithi022 podman[112148]: 2024-03-24 04:36:02.805996645 +0000 UTC m=+0.624311233 container died 7297ea630096102f6f29cd8d9b972e26c15664644d4c5a834f1ad0dd7467fef4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T04:36:03.392 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T04:36:03.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:03.438 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T04:36:03.438 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T04:36:03.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.3 2024-03-24T04:36:03.792 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:03 smithi022 systemd[1]: Stopping Ceph osd.3 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:36:04.047 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:36:03.791+0000 7f80d16d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:36:04.047 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:36:03.791+0000 7f80d16d9700 -1 osd.3 176 *** Got signal Terminated *** 2024-03-24T04:36:04.048 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:03 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:36:03.791+0000 7f80d16d9700 -1 osd.3 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:05.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:05 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:05.230+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:06.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:06 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:06.304+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:06.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:06 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:06.182+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:07.297 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:06 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:36:06.962+0000 7f80cdccd700 -1 osd.3 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.798172+0000 front 2024-03-24T04:35:40.798192+0000 (oldest deadline 2024-03-24T04:36:06.698328+0000) 2024-03-24T04:36:07.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:07 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:07.274+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:07.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:07 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:07.175+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:08.140 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:07 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:07.973+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:08.140 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:07 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:07.732+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:08.297 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:08 smithi022 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3[80265]: 2024-03-24T04:36:08.002+0000 7f80cdccd700 -1 osd.3 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.798172+0000 front 2024-03-24T04:35:40.798192+0000 (oldest deadline 2024-03-24T04:36:06.698328+0000) 2024-03-24T04:36:08.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:08.301+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:08.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:08.139+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:09.146 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:08.942+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:09.146 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:08 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:08.752+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:09.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:09 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:09.318+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:09.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:09 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:09.146+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:09.658 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:09 smithi022 podman[112251]: 2024-03-24 04:36:09.399143957 +0000 UTC m=+5.732788432 container died 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T04:36:09.933 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:09 smithi022 podman[112251]: 2024-03-24 04:36:09.732480428 +0000 UTC m=+6.066124903 container cleanup 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T04:36:09.934 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:09 smithi022 bash[112251]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3 2024-03-24T04:36:10.100 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:09 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:09.913+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:10.100 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:09 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:09.771+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:10.297 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:09 smithi022 podman[112301]: 2024-03-24 04:36:09.933913416 +0000 UTC m=+0.539497590 container remove 0a452e1436b9e72ac28765da1b260a87d09fc2e0de281046df74079e5f6bc58a (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True) 2024-03-24T04:36:10.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:10 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:10.332+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:10.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:10 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:10.100+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:10.603 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.257304788 +0000 UTC m=+0.018548515 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:10.604 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.3597014 +0000 UTC m=+0.120945126 container create 3d444cef5c906e9a6a86402c184ab7f9e5da50ab6f1e5feb34730f95583818bf (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T04:36:10.604 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.604013086 +0000 UTC m=+0.365256824 container init 3d444cef5c906e9a6a86402c184ab7f9e5da50ab6f1e5feb34730f95583818bf (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T04:36:10.906 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.60761172 +0000 UTC m=+0.368855449 container start 3d444cef5c906e9a6a86402c184ab7f9e5da50ab6f1e5feb34730f95583818bf (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T04:36:10.906 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.696067342 +0000 UTC m=+0.457311068 container attach 3d444cef5c906e9a6a86402c184ab7f9e5da50ab6f1e5feb34730f95583818bf (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:11.081 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:10 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:10.885+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:11.081 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:10 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:10.769+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:11.207 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 04:36:10 smithi022 podman[112398]: 2024-03-24 04:36:10.90695246 +0000 UTC m=+0.668196209 container died 3d444cef5c906e9a6a86402c184ab7f9e5da50ab6f1e5feb34730f95583818bf (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T04:36:11.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:11.324+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:11.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:11.080+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:11.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T04:36:11.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:11.528 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T04:36:11.528 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T04:36:11.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.4 2024-03-24T04:36:11.857 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:11 smithi116 systemd[1]: Stopping Ceph osd.4 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:36:11.857 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:11.778+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:12.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:12.032+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:12.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:11.857+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:12.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:11.865+0000 7f24fbc0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:36:12.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:11.865+0000 7f24fbc0b700 -1 osd.4 176 *** Got signal Terminated *** 2024-03-24T04:36:12.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:11 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:11.865+0000 7f24fbc0b700 -1 osd.4 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:12.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:12.349+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:13.062 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:12.838+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:13.062 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:12 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:12.774+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:13.397 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:13 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:13.061+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:13.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:13 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:13.395+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:14.223 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:13 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:13.794+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:14.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:13 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:13.807+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:14.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:13 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:13.807+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:14.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:14.017+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:14.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:14.017+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:14.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:14.427+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:15.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:14.978+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:15.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:14.978+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:15.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:14.844+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:15.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:14.844+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:49.976159+0000 front 2024-03-24T04:35:49.976121+0000 (oldest deadline 2024-03-24T04:36:14.075985+0000) 2024-03-24T04:36:15.225 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:14.815+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:15.225 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:14 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:14.815+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:15.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:15 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:15.460+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:16.223 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:15 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:15.868+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:16.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:15 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:15.868+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:49.976159+0000 front 2024-03-24T04:35:49.976121+0000 (oldest deadline 2024-03-24T04:36:14.075985+0000) 2024-03-24T04:36:16.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:16.002+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:16.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:16.002+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:16.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:15 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:15.847+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:16.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:15 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:15.847+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:16.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:16.413+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:17.223 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:16.827+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.575005+0000 front 2024-03-24T04:35:43.575100+0000 (oldest deadline 2024-03-24T04:36:07.075182+0000) 2024-03-24T04:36:17.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4[58137]: 2024-03-24T04:36:16.827+0000 7f24f81ff700 -1 osd.4 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:49.976159+0000 front 2024-03-24T04:35:49.976121+0000 (oldest deadline 2024-03-24T04:36:14.075985+0000) 2024-03-24T04:36:17.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:17.027+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:17.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:17.027+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:17.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:16.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:17.225 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:16 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:16.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:17.667 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:17 smithi116 podman[78893]: 2024-03-24 04:36:17.379839847 +0000 UTC m=+5.658300445 container died 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T04:36:17.667 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:17.402+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:17.667 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:17.402+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:17.973 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:17 smithi116 podman[78893]: 2024-03-24 04:36:17.849308314 +0000 UTC m=+6.127768966 container cleanup 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T04:36:17.973 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:17 smithi116 bash[78893]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4 2024-03-24T04:36:17.973 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:17.913+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:17.974 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:17 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:17.913+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:18.393 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[78910]: 2024-03-24 04:36:18.110470477 +0000 UTC m=+0.729828082 container remove 97425f93c8654f1308f0542f60f7a44bcaafbd7cdaab5777e1ce825bdb7fd11b (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T04:36:18.393 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:18.045+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:18.393 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:18.045+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:18.704 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:18.392+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:18.704 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:18.392+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:18.704 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[79001]: 2024-03-24 04:36:18.36405803 +0000 UTC m=+0.017201205 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:18.704 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[79001]: 2024-03-24 04:36:18.468404906 +0000 UTC m=+0.121548080 container create cf967fa69601098aa048f8ce2ab9fdcf7447180c47c7e039fd36111d8fa84bc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T04:36:18.974 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[79001]: 2024-03-24 04:36:18.704287656 +0000 UTC m=+0.357430834 container init cf967fa69601098aa048f8ce2ab9fdcf7447180c47c7e039fd36111d8fa84bc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T04:36:18.974 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[79001]: 2024-03-24 04:36:18.708447124 +0000 UTC m=+0.361590302 container start cf967fa69601098aa048f8ce2ab9fdcf7447180c47c7e039fd36111d8fa84bc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T04:36:18.975 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:18 smithi116 podman[79001]: 2024-03-24 04:36:18.787652403 +0000 UTC m=+0.440795595 container attach cf967fa69601098aa048f8ce2ab9fdcf7447180c47c7e039fd36111d8fa84bc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:36:18.975 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:18.897+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:18.975 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:18 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:18.897+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:19.282 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 04:36:19 smithi116 podman[79001]: 2024-03-24 04:36:19.002724311 +0000 UTC m=+0.655867486 container died cf967fa69601098aa048f8ce2ab9fdcf7447180c47c7e039fd36111d8fa84bc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:19.282 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:19.002+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:19.282 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:19.002+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:19.574 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:19.393+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:19.574 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:19.393+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:19.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T04:36:19.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:19.632 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T04:36:19.633 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T04:36:19.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.5 2024-03-24T04:36:19.887 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 systemd[1]: Stopping Ceph osd.5 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:36:20.194 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:20.047+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:20.194 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:20.047+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:20.194 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:19.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:20.194 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:19.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:20.194 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:19.985+0000 7f99a0e66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:36:20.195 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:19.985+0000 7f99a0e66700 -1 osd.5 176 *** Got signal Terminated *** 2024-03-24T04:36:20.195 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:19 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:19.985+0000 7f99a0e66700 -1 osd.5 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:20.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:20.430+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:20.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:20.430+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:21.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:20.840+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:21.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:20 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:20.840+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:21.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:21.001+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:21.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:21.001+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:21.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:21.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:21.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:21.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:22.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:21.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:22.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:21 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:21.886+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:22.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:22.041+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:22.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:22.041+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:22.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:22.387+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:22.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:22.387+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:22.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:22.387+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:23.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:22.901+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:23.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:22.901+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:23.224 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:22 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:22.901+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.665000+0000 front 2024-03-24T04:35:58.664982+0000 (oldest deadline 2024-03-24T04:36:22.764862+0000) 2024-03-24T04:36:23.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:23.083+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:23.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:23.083+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:23.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:23.343+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:23.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:23.343+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:23.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:23.343+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:24.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:23.900+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:24.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:23.900+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:24.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:23 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:23.900+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.665000+0000 front 2024-03-24T04:35:58.664982+0000 (oldest deadline 2024-03-24T04:36:22.764862+0000) 2024-03-24T04:36:24.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:24.131+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:24.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:24.131+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:24.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:24.131+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:24.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:24.372+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:24.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:24.372+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:24.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:24.372+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:25.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:24.857+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.463270+0000 front 2024-03-24T04:35:43.463286+0000 (oldest deadline 2024-03-24T04:36:07.563179+0000) 2024-03-24T04:36:25.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:24.857+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.664185+0000 front 2024-03-24T04:35:48.664242+0000 (oldest deadline 2024-03-24T04:36:13.364179+0000) 2024-03-24T04:36:25.223 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:24 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5[60399]: 2024-03-24T04:36:24.857+0000 7f999d45a700 -1 osd.5 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.665000+0000 front 2024-03-24T04:35:58.664982+0000 (oldest deadline 2024-03-24T04:36:22.764862+0000) 2024-03-24T04:36:25.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:25.101+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:25.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:25.101+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:25.224 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:25.101+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:25.723 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:25 smithi116 podman[79119]: 2024-03-24 04:36:25.485312972 +0000 UTC m=+5.603727820 container died bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T04:36:25.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:25.338+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:25.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:25.338+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:25.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:25 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:25.338+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:26.075 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:25 smithi116 podman[79119]: 2024-03-24 04:36:25.935682044 +0000 UTC m=+6.054096890 container cleanup bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T04:36:26.075 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:25 smithi116 bash[79119]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5 2024-03-24T04:36:26.473 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79137]: 2024-03-24 04:36:26.237469322 +0000 UTC m=+0.750148275 container remove bb9dd19044c4f94be1c06d28fe9aded13580689d885a90f0ca37a30130d9ede5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:26.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:26.074+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:26.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:26.074+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:26.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:26.074+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:26.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:26.322+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:26.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:26.322+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:26.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:26 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:26.322+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:26.798 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79226]: 2024-03-24 04:36:26.448894164 +0000 UTC m=+0.017786981 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:26.799 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79226]: 2024-03-24 04:36:26.588493869 +0000 UTC m=+0.157386663 container create 7859078ea1c89fc34222a0ee1e4a835fccb02d2a4f4c84d231eeae13beb1d0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T04:36:26.799 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79226]: 2024-03-24 04:36:26.799193541 +0000 UTC m=+0.368086358 container init 7859078ea1c89fc34222a0ee1e4a835fccb02d2a4f4c84d231eeae13beb1d0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T04:36:27.088 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79226]: 2024-03-24 04:36:26.802487805 +0000 UTC m=+0.371380602 container start 7859078ea1c89fc34222a0ee1e4a835fccb02d2a4f4c84d231eeae13beb1d0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T04:36:27.088 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:26 smithi116 podman[79226]: 2024-03-24 04:36:26.874008875 +0000 UTC m=+0.442901669 container attach 7859078ea1c89fc34222a0ee1e4a835fccb02d2a4f4c84d231eeae13beb1d0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:27.443 INFO:journalctl@ceph.osd.5.smithi116.stdout:Mar 24 04:36:27 smithi116 podman[79226]: 2024-03-24 04:36:27.088592831 +0000 UTC m=+0.657485634 container died 7859078ea1c89fc34222a0ee1e4a835fccb02d2a4f4c84d231eeae13beb1d0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T04:36:27.443 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:27.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:27.443 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:27.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:27.444 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:27.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:27.444 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:27.098+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:27.444 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:27.098+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:27.444 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:27 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:27.098+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:27.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T04:36:27.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:27.771 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T04:36:27.771 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T04:36:27.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.6 2024-03-24T04:36:27.973 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:27 smithi116 systemd[1]: Stopping Ceph osd.6 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:28.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:28.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:28.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.088+0000 7f40fe14d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.088+0000 7f40fe14d700 -1 osd.6 176 *** Got signal Terminated *** 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.088+0000 7f40fe14d700 -1 osd.6 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:28.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.126+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:28.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.126+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:28.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:28 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:28.126+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:29.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:29.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:29.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:29.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:29.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:29.296+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:29.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:29.176+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:29.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:29.176+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:29.474 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:29 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:29.176+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:30.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:30.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:30.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:30.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:30.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:30.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:07.935169+0000 front 2024-03-24T04:36:07.935219+0000 (oldest deadline 2024-03-24T04:36:29.635010+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:30.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:30.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:30.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:30.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:30 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:30.326+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:31.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:31.204+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:31.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:31.204+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:31.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:31.204+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:31.473 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:31.204+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:07.935169+0000 front 2024-03-24T04:36:07.935219+0000 (oldest deadline 2024-03-24T04:36:29.635010+0000) 2024-03-24T04:36:31.473 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:31.286+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:31.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:31.286+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:31.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:31.286+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:31.474 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:31 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:31.286+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:32.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:43.432342+0000 front 2024-03-24T04:35:43.432299+0000 (oldest deadline 2024-03-24T04:36:05.132127+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:32.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:48.633056+0000 front 2024-03-24T04:35:48.633072+0000 (oldest deadline 2024-03-24T04:36:13.332958+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:32.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:58.534327+0000 front 2024-03-24T04:35:58.534295+0000 (oldest deadline 2024-03-24T04:36:23.834211+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6[62659]: 2024-03-24T04:36:32.224+0000 7f40f6739700 -1 osd.6 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:07.935169+0000 front 2024-03-24T04:36:07.935219+0000 (oldest deadline 2024-03-24T04:36:29.635010+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:32.333+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:32.333+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:32.333+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:32.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:32 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:32.333+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:33.723 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:33 smithi116 podman[79365]: 2024-03-24 04:36:33.469997187 +0000 UTC m=+5.475136773 container died 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T04:36:33.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:33.379+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:33.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:33.379+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:33.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:33.379+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:33.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:33 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:33.379+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:34.072 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:33 smithi116 podman[79365]: 2024-03-24 04:36:33.821987905 +0000 UTC m=+5.827127491 container cleanup 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T04:36:34.072 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:33 smithi116 bash[79365]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6 2024-03-24T04:36:34.382 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79383]: 2024-03-24 04:36:34.073049161 +0000 UTC m=+0.598295365 container remove 7d40be1a1ff8940fe483a6361729221fdb4964fa369ccef3345b26d8b2e46714 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T04:36:34.668 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.382356102 +0000 UTC m=+0.106592682 container create f3a628f408f1dd767d8f3ccdd2da534fb1502cd82f000d464f0b9c5a3ea71dce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:34.668 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.294157667 +0000 UTC m=+0.018394263 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:34.668 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.668122162 +0000 UTC m=+0.392358762 container init f3a628f408f1dd767d8f3ccdd2da534fb1502cd82f000d464f0b9c5a3ea71dce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T04:36:34.668 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:34 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:34.397+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:34.669 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:34 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:34.397+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:34.669 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:34 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:34.397+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:34.669 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:34 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:34.397+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:34.966 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.671840152 +0000 UTC m=+0.396076733 container start f3a628f408f1dd767d8f3ccdd2da534fb1502cd82f000d464f0b9c5a3ea71dce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T04:36:34.967 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.759760809 +0000 UTC m=+0.483997389 container attach f3a628f408f1dd767d8f3ccdd2da534fb1502cd82f000d464f0b9c5a3ea71dce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:35.223 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 04:36:34 smithi116 podman[79472]: 2024-03-24 04:36:34.967163763 +0000 UTC m=+0.691400367 container died f3a628f408f1dd767d8f3ccdd2da534fb1502cd82f000d464f0b9c5a3ea71dce (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T04:36:35.512 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.437+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:35.513 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.437+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:35.513 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.437+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:35.513 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.437+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:35.531 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T04:36:35.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:35.574 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T04:36:35.574 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T04:36:35.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7 2024-03-24T04:36:35.891 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 systemd[1]: Stopping Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715... 2024-03-24T04:36:36.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.890+0000 7f5c6be97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T04:36:36.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.890+0000 7f5c6be97700 -1 osd.7 176 *** Got signal Terminated *** 2024-03-24T04:36:36.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:35 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:35.890+0000 7f5c6be97700 -1 osd.7 176 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T04:36:36.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:36.459+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:36.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:36.459+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:36.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:36.459+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:36.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:36 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:36.459+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:37.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:37.431+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:37.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:37.432+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:37.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:37.432+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:37.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:37 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:37.432+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:38.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:38 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:38.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:38.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:38 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:38.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:38.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:38 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:38.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:38.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:38 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:38.383+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:39.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:39 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:39.412+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:39.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:39 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:39.412+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:39.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:39 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:39.412+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:39.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:39 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:39.412+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:40.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:40 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:40.421+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T04:35:40.944641+0000 front 2024-03-24T04:35:40.944679+0000 (oldest deadline 2024-03-24T04:36:06.244685+0000) 2024-03-24T04:36:40.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:40 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:40.421+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6814 osd.1 since back 2024-03-24T04:35:52.546678+0000 front 2024-03-24T04:35:52.546668+0000 (oldest deadline 2024-03-24T04:36:16.646581+0000) 2024-03-24T04:36:40.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:40 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:40.421+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6822 osd.2 since back 2024-03-24T04:35:56.647152+0000 front 2024-03-24T04:35:56.647203+0000 (oldest deadline 2024-03-24T04:36:21.947074+0000) 2024-03-24T04:36:40.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:40 smithi116 ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7[64918]: 2024-03-24T04:36:40.421+0000 7f5c64483700 -1 osd.7 176 heartbeat_check: no reply from 172.21.15.22:6830 osd.3 since back 2024-03-24T04:36:08.348575+0000 front 2024-03-24T04:36:08.348551+0000 (oldest deadline 2024-03-24T04:36:30.048423+0000) 2024-03-24T04:36:41.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:41 smithi116 podman[79576]: 2024-03-24 04:36:41.484509841 +0000 UTC m=+5.687458980 container died a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T04:36:42.107 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:41 smithi116 podman[79576]: 2024-03-24 04:36:41.848622859 +0000 UTC m=+6.051572012 container cleanup a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-24T04:36:42.108 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:41 smithi116 bash[79576]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7 2024-03-24T04:36:42.450 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79607]: 2024-03-24 04:36:42.108170164 +0000 UTC m=+0.618275956 container remove a8a47b2d03e523f7812fd6122e62f39a0da95ea0c3458d87db7423a5d4978d58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T04:36:42.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.450506241 +0000 UTC m=+0.109468773 container create 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T04:36:42.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.359180131 +0000 UTC m=+0.018142680 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T04:36:42.723 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.653261392 +0000 UTC m=+0.312223931 container init 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T04:36:42.724 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.656236104 +0000 UTC m=+0.315198661 container start 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:36:43.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.761674008 +0000 UTC m=+0.420636558 container attach 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T04:36:43.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:42 smithi116 podman[79699]: 2024-03-24 04:36:42.945573247 +0000 UTC m=+0.604535781 container died 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T04:36:43.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T04:36:43.533 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:43 smithi116 podman[79725]: 2024-03-24 04:36:43.481484819 +0000 UTC m=+0.526863617 container remove 13d328a3f31856254083bcf1e59eb8f53fe7689d6ac83b70f4b3a6e06afee118 (image=quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-8f0be64c-e994-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T04:36:43.533 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:43 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-24T04:36:43.534 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:43 smithi116 systemd[1]: Stopped Ceph osd.7 for 8f0be64c-e994-11ee-95cd-87774f69a715. 2024-03-24T04:36:43.534 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 04:36:43 smithi116 systemd[1]: ceph-8f0be64c-e994-11ee-95cd-87774f69a715@osd.7.service: Consumed 15.738s CPU time. 2024-03-24T04:36:43.550 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T04:36:43.550 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T04:36:43.550 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T04:39:39.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T04:39:58.894 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:39:58.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T04:39:58.961 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T04:39:58.964 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi022/crash 2024-03-24T04:39:58.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash -- . 2024-03-24T04:39:59.016 INFO:teuthology.orchestra.run.smithi022.stderr:tar: /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T04:39:59.016 INFO:teuthology.orchestra.run.smithi022.stderr:tar: Error is not recoverable: exiting now 2024-03-24T04:39:59.018 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi116/crash 2024-03-24T04:39:59.019 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash -- . 2024-03-24T04:39:59.067 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T04:39:59.068 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-24T04:39:59.068 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T04:39:59.069 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T04:39:59.100 INFO:teuthology.orchestra.run.smithi022.stdout:2024-03-24T04:30:00.000163+0000 mon.smithi022 (mon.0) 543 : cluster 3 [WRN] OSD_DOWN: 1 osds down 2024-03-24T04:39:59.100 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T04:39:59.100 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[SEC\]' /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T04:39:59.170 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]' /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T04:39:59.241 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[WRN\]' /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T04:39:59.310 INFO:teuthology.orchestra.run.smithi022.stdout:2024-03-24T04:30:00.000163+0000 mon.smithi022 (mon.0) 543 : cluster 3 [WRN] OSD_DOWN: 1 osds down 2024-03-24T04:39:59.311 INFO:tasks.cephadm:Compressing logs... 2024-03-24T04:39:59.311 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T04:39:59.354 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T04:39:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T04:39:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:: No such file or directory 2024-03-24T04:39:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi022.ekdqmp.log 2024-03-24T04:39:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log/var/log/ceph/cephadm.log: 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi022.ijpjvk.log 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi022.ekdqmp.log: /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:gzip/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi022.ijpjvk.log: -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi022.log 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi022.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T04:39:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T04:39:59.383 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T04:39:59.383 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-24T04:39:59.383 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T04:39:59.383 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi116.oxktjt.log 2024-03-24T04:39:59.383 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T04:39:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.0.log: 91.5% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T04:39:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log 2024-03-24T04:39:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T04:39:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T04:39:59.392 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log: 87.4% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T04:39:59.392 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T04:39:59.393 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi116.oxktjt.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi116.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi116.log: gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.5.log: -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi116.kvgpsf.log 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log: 91.6% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T04:39:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T04:39:59.395 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi116.kvgpsf.log: 88.6% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi116.kvgpsf.log.gz 2024-03-24T04:39:59.395 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T04:39:59.395 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log: 87.4% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T04:39:59.397 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.2.log: 89.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T04:39:59.404 INFO:teuthology.orchestra.run.smithi022.stderr: 91.4% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mds.foofs.smithi022.ekdqmp.log.gz 2024-03-24T04:39:59.404 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T04:39:59.422 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log: 91.3% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T04:39:59.443 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi116.oxktjt.log.gz 2024-03-24T04:39:59.657 INFO:teuthology.orchestra.run.smithi022.stderr: 88.2% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mgr.smithi022.ijpjvk.log.gz 2024-03-24T04:39:59.857 INFO:teuthology.orchestra.run.smithi116.stderr: 92.6% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi116.log.gz 2024-03-24T04:40:00.838 INFO:teuthology.orchestra.run.smithi022.stderr: 92.0% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-mon.smithi022.log.gz 2024-03-24T04:40:14.095 INFO:teuthology.orchestra.run.smithi116.stderr: 93.9% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T04:40:15.360 INFO:teuthology.orchestra.run.smithi022.stderr: 94.3% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T04:40:16.943 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T04:40:17.909 INFO:teuthology.orchestra.run.smithi022.stderr: 94.0% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T04:40:19.137 INFO:teuthology.orchestra.run.smithi116.stderr: 94.1% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T04:40:19.928 INFO:teuthology.orchestra.run.smithi022.stderr: 94.0% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T04:40:20.562 INFO:teuthology.orchestra.run.smithi022.stderr: 94.0% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T04:40:20.564 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T04:40:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:real 0m21.198s 2024-03-24T04:40:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:user 1m15.192s 2024-03-24T04:40:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m2.730s 2024-03-24T04:40:23.481 INFO:teuthology.orchestra.run.smithi116.stderr: 93.9% -- replaced with /var/log/ceph/8f0be64c-e994-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T04:40:23.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-24T04:40:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m24.115s 2024-03-24T04:40:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:user 1m13.577s 2024-03-24T04:40:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m2.969s 2024-03-24T04:40:23.485 INFO:tasks.cephadm:Archiving logs... 2024-03-24T04:40:23.485 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi022/log 2024-03-24T04:40:23.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T04:40:32.993 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi116/log 2024-03-24T04:40:32.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T04:40:42.573 INFO:tasks.cephadm:Removing cluster... 2024-03-24T04:40:42.573 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 --force 2024-03-24T04:40:42.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8f0be64c-e994-11ee-95cd-87774f69a715 --force 2024-03-24T04:40:43.347 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T04:40:43.347 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T04:40:43.363 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T04:40:43.378 INFO:tasks.cephadm:Teardown complete 2024-03-24T04:40:43.379 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T04:40:43.389 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T04:40:43.389 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T04:40:43.406 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T04:40:43.418 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +129us[ +129us] +/- 87ms 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -342us[ -299us] +/- 79ms 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +293us[ +293us] +/- 69ms 2024-03-24T04:40:43.423 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:40:43.432 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +94us[ +94us] +/- 87ms 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 14 -367us[ -350us] +/- 79ms 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv03.front.sepia.ceph.com 3 6 377 17 +285us[ +302us] +/- 69ms 2024-03-24T04:40:43.437 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T04:40:43.437 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T04:40:43.449 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T04:40:43.450 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T04:40:43.464 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T04:40:43.548 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T04:40:43.625 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T04:40:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:40:43.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T04:40:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:40:43.687 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T04:40:43.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T04:40:43.713 INFO:teuthology.task.internal:Duration was 2020.114633 seconds 2024-03-24T04:40:43.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T04:40:43.729 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T04:40:43.729 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T04:40:43.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T04:40:43.772 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T04:40:43.774 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T04:40:44.088 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T04:40:44.088 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T04:40:44.089 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T04:40:44.148 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T04:40:44.149 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T04:40:44.185 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T04:40:44.186 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T04:40:44.192 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T04:40:44.254 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T04:40:44.254 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T04:40:44.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T04:40:58.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T04:40:58.590 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T04:40:58.590 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T04:40:58.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T04:40:58.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T04:40:58.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T04:40:58.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T04:40:58.686 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-24T04:40:58.721 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-24T04:40:59.048 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T04:40:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:40:59.063 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T04:40:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T04:40:59.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T04:40:59.117 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T04:40:59.118 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi022 2024-03-24T04:40:59.118 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T04:40:59.192 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847/remote/smithi116 2024-03-24T04:40:59.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T04:40:59.240 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T04:40:59.240 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T04:40:59.243 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T04:40:59.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T04:40:59.301 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T04:40:59.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T04:40:59.313 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T04:40:59.313 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T04:40:59.316 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T04:40:59.327 INFO:teuthology.orchestra.run.smithi022.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 04:40 /home/ubuntu/cephtest 2024-03-24T04:40:59.346 INFO:teuthology.orchestra.run.smithi116.stdout: 265622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 04:40 /home/ubuntu/cephtest 2024-03-24T04:40:59.347 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T04:40:59.360 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T04:40:59.388 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T04:40:59.418 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-24T04:40:59.418 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T04:40:59.441 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-24T04:40:59.441 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T04:40:59.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 03:58:06.248333', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOdWTIXrRM+iCeSvS9IDw2aoJZ06gm3sWkuonouXzH6r5kkQbWIfcve8AKmLA76lBtIMmJOhOUA/gCU18dN0l+o='} 2024-03-24T04:40:59.468 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 03:58:06.249706', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqLqfO5Av72MO/1zBNgmF7F6/oz+SmYEXyi1RqQphw95A71H44jESKZrmdCF624G49Bqj9/emWZXbQP9EtOkcs='} 2024-03-24T04:40:59.491 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-24T04:40:59.491 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T04:40:59.516 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-24T04:40:59.517 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:40:59.532 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-24T04:40:59.532 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T04:40:59.540 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T04:40:59.554 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-24T04:40:59.554 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:40:59.577 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T04:41:03.541 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:41:03.578 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:41:03.653 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T04:41:03.688 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T04:41:07.659 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:41:07.689 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:41:07.774 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-24T04:41:07.800 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T04:41:07.875 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-24T04:41:11.802 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T04:41:11.912 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-24T04:41:12.013 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-24T04:41:12.116 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2020.114632844925 failure_reason: '"2024-03-24T04:30:00.000163+0000 mon.smithi022 (mon.0) 543 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T04:41:12.116 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T04:41:12.200 INFO:teuthology.run:FAIL