2024-04-06T03:01:31.899 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-06T03:01:31.899 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915 --name teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/orig.config.yaml 2024-04-06T03:01:31.936 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T03:01:32.028 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7641915' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 5609 sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 sleep_before_teardown: 0 subset: 32/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGJE9dFfyLhLYJZCyfNuJuhCTRz2BVsRNYM1v8ZM+nydf1kdjmh0kncdf30Sly5H99ubXZyoJ32SaNyikF2QQ0= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJwL8gk2Il6mUZCvxSVovtI4ZIlAuhXbjaSBaebCdfqB7YXK2MUM6vbQ8QrDI/Rc6DJwGhJbpvFUvxrVpf1uEY= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-06T03:01:32.028 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa; will attempt to use it 2024-04-06T03:01:32.029 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks 2024-04-06T03:01:32.029 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T03:01:32.031 INFO:teuthology.task.internal:Checking packages... 2024-04-06T03:01:32.055 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6' 2024-04-06T03:01:32.055 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T03:01:32.055 INFO:teuthology.packaging:ref: None 2024-04-06T03:01:32.055 INFO:teuthology.packaging:tag: None 2024-04-06T03:01:32.055 INFO:teuthology.packaging:branch: squid 2024-04-06T03:01:32.055 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:01:32.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-06T03:01:32.272 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2027.g9f192350 2024-04-06T03:01:32.273 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T03:01:32.282 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T03:01:32.282 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T03:01:32.287 INFO:teuthology.task.internal:Saving configuration 2024-04-06T03:01:32.300 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T03:01:32.306 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T03:01:32.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:56:38.975483', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGJE9dFfyLhLYJZCyfNuJuhCTRz2BVsRNYM1v8ZM+nydf1kdjmh0kncdf30Sly5H99ubXZyoJ32SaNyikF2QQ0='} 2024-04-06T03:01:32.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:56:38.976930', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJwL8gk2Il6mUZCvxSVovtI4ZIlAuhXbjaSBaebCdfqB7YXK2MUM6vbQ8QrDI/Rc6DJwGhJbpvFUvxrVpf1uEY='} 2024-04-06T03:01:32.357 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T03:01:32.363 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-06T03:01:32.363 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-06T03:01:32.363 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T03:01:32.433 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6a9f58cf70>, signals=[15]) 2024-04-06T03:01:32.434 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T03:01:32.440 INFO:teuthology.task.internal:Opening connections... 2024-04-06T03:01:32.440 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:01:32.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:01:32.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-04-06T03:01:32.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:01:32.638 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T03:01:32.644 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-04-06T03:01:32.663 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-04-06T03:01:32.663 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-04-06T03:01:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-04-06T03:01:32.722 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T03:01:32.722 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-04-06T03:01:32.722 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T03:01:32.722 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T03:01:32.722 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T03:01:32.722 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-04-06T03:01:32.742 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-04-06T03:01:32.759 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-04-06T03:01:32.759 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T03:01:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T03:01:32.817 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T03:01:32.817 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-04-06T03:01:32.842 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T03:01:32.851 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T03:01:32.857 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T03:01:32.857 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-04-06T03:01:32.860 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-04-06T03:01:32.875 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T03:01:32.881 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T03:01:32.881 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-04-06T03:01:32.918 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-04-06T03:01:32.939 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T03:01:32.990 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T03:01:33.005 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-06T03:01:33.005 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-06T03:01:33.006 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T03:01:33.006 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T03:01:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:01:33.023 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-04-06T03:01:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:01:33.024 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-04-06T03:01:33.081 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-361.el9.x86_64 2024-04-06T03:01:33.081 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-361.el9.x86_64 2024-04-06T03:01:33.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-04-06T03:01:33.082 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-04-06T03:01:33.082 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2024-04-06T03:01:33.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-04-06T03:01:35.278 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-06T03:01:35.309 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-06T03:01:36.984 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 8.3 MB/s | 2.5 MB 00:00 2024-04-06T03:01:37.105 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.5 MB 00:00 2024-04-06T03:01:38.227 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-06T03:01:44.591 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-04-06T03:01:48.733 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-04-06T03:01:52.801 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-04-06T03:01:54.543 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-04-06T03:01:55.318 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-06T03:01:56.713 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-06T03:01:59.902 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.1 MB 00:01 2024-04-06T03:02:01.527 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-06T03:02:01.912 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-06T03:02:03.104 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-06T03:02:04.134 INFO:teuthology.orchestra.run.smithi060.stderr:Modular dependency problems: 2024-04-06T03:02:04.134 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:02:04.134 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 1: conflicting requests 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 2: conflicting requests 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 3: conflicting requests 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 4: conflicting requests 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 5: conflicting requests 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.135 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 6: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 7: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 8: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 9: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 10: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 11: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 12: conflicting requests 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.136 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 13: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 14: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 15: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 16: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 17: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 18: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 19: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 20: conflicting requests 2024-04-06T03:02:04.137 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 21: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 22: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 23: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 24: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 25: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 26: conflicting requests 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.138 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 27: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 28: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 29: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 30: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 31: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 32: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 33: conflicting requests 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.139 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 34: conflicting requests 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 35: conflicting requests 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 36: conflicting requests 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 37: conflicting requests 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.140 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 39: conflicting requests 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.141 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.142 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.143 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.144 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T03:02:04.594 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T03:02:04.595 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T03:02:04.696 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-06T03:02:04.707 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:02:04.708 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 76 M 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 123 M 2024-04-06T03:02:04.709 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-06T03:02:05.013 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 347 kB/s | 66 kB 00:00 2024-04-06T03:02:08.196 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-06T03:02:08.313 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.3 MB/s | 15 MB 00:03 2024-04-06T03:02:08.580 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 7.7 MB/s | 27 MB 00:03 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr:Modular dependency problems: 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 1: conflicting requests 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 2: conflicting requests 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.336 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 3: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 4: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 5: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 6: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 7: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 8: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 9: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 10: conflicting requests 2024-04-06T03:02:10.337 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 11: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 12: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 13: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 14: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 15: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 16: conflicting requests 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.338 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 17: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 18: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 19: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 20: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 21: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 22: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 23: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 24: conflicting requests 2024-04-06T03:02:10.339 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 25: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 26: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 27: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 28: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 29: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 30: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 31: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 32: conflicting requests 2024-04-06T03:02:10.340 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 33: conflicting requests 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 34: conflicting requests 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 35: conflicting requests 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 36: conflicting requests 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 37: conflicting requests 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T03:02:10.341 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 39: conflicting requests 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.342 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.343 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.344 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T03:02:10.778 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T03:02:10.778 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T03:02:10.908 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T03:02:10.924 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 76 M 2024-04-06T03:02:10.925 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-04-06T03:02:10.926 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-04-06T03:02:11.397 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 197 kB/s | 66 kB 00:00 2024-04-06T03:02:15.838 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 3.0 MB/s | 33 MB 00:11 2024-04-06T03:02:15.839 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-06T03:02:15.839 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.8 MB/s | 76 MB 00:11 2024-04-06T03:02:15.880 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.1 MB/s | 15 MB 00:04 2024-04-06T03:02:16.534 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-06T03:02:16.963 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-06T03:02:16.963 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-06T03:02:17.113 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 5.5 MB/s | 33 MB 00:06 2024-04-06T03:02:17.974 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-06T03:02:17.974 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-06T03:02:18.255 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 4.0 MB/s | 27 MB 00:06 2024-04-06T03:02:18.255 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-04-06T03:02:18.255 INFO:teuthology.orchestra.run.smithi195.stdout:Total 10 MB/s | 76 MB 00:07 2024-04-06T03:02:18.919 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-04-06T03:02:19.282 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-04-06T03:02:19.282 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-04-06T03:02:20.306 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-04-06T03:02:20.307 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-04-06T03:02:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-06T03:02:21.200 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T03:02:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:02:22.746 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-04-06T03:02:23.422 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T03:02:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:02:24.173 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:02:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:02:26.255 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:02:26.390 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:02:29.391 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:02:29.428 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:02:31.100 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:02:31.218 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:02:34.526 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:02:35.955 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.257 INFO:teuthology.orchestra.run.smithi195.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T03:03:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:03:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.469 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.469 INFO:teuthology.orchestra.run.smithi060.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T03:03:05.469 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:03:05.515 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T03:03:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:03:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:03:05.858 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:05.858 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T03:03:05.858 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T03:03:05.858 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T03:03:06.125 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:06.125 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:03:06.126 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.301 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:03:06.301 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-06T03:03:06.452 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T03:03:06.563 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.564 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.564 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.564 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T03:03:06.564 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T03:03:06.564 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-06T03:03:06.564 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T03:03:06.564 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T03:03:06.645 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.646 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-04-06T03:03:06.647 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-04-06T03:03:06.681 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-04-06T03:03:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:03:06.683 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T03:03:06.684 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T03:03:06.684 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T03:03:06.805 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.806 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.806 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.806 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T03:03:06.807 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T03:03:06.807 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-06T03:03:06.807 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T03:03:06.807 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T03:03:06.890 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T03:03:06.890 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-04-06T03:03:06.928 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-04-06T03:03:06.928 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-04-06T03:03:06.928 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-04-06T03:03:06.928 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-04-06T03:03:06.928 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T03:03:06.929 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T03:03:06.930 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T03:03:06.930 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-04-06T03:03:06.930 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:03:06.930 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T03:03:06.931 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T03:03:06.931 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-432.el9.x86_64 2024-04-06T03:03:06.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T03:03:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-04-06T03:03:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-04-06T03:03:08.743 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T03:03:08.782 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-04-06T03:03:08.785 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-04-06T03:03:08.800 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.Vt8Th6tCBx 2024-04-06T03:03:08.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Vt8Th6tCBx 2024-04-06T03:03:08.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.Vt8Th6tCBx 2024-04-06T03:03:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T03:03:09.011 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.Vt8Th6tCBx is 6KB 2024-04-06T03:03:09.015 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-04-06T03:03:09.018 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-04-06T03:03:09.031 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.Vt8Th6tCBx 2024-04-06T03:03:09.033 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.bb76zvQWH0 2024-04-06T03:03:09.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bb76zvQWH0 2024-04-06T03:03:09.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T03:03:09.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.bb76zvQWH0 2024-04-06T03:03:09.114 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T03:03:09.114 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T03:03:09.114 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T03:03:09.114 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T03:03:09.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T03:03:09.245 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.bb76zvQWH0 is 6KB 2024-04-06T03:03:09.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-04-06T03:03:09.294 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.bb76zvQWH0 2024-04-06T03:03:09.309 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T03:03:09.377 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T03:03:09.377 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T03:03:09.377 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T03:03:09.377 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T03:03:09.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T03:03:09.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-04-06T03:03:39.297 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-06T03:03:39.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-06T03:03:39.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:03:39.539 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-06T03:03:39.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:03:39.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:04:14.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-04-06T03:04:23.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:04:23.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:04:26.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-04-06T03:04:38.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:04:38.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:04:39.301 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T03:04:39.549 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-04-06T03:04:39.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:04:39.951 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-04-06T03:04:39.951 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-04-06T03:04:39.966 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-432.el9.x86_64 2024-04-06T03:04:39.967 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T03:04:39.967 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T03:04:39.967 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T03:04:40.967 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-06T03:04:40.967 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T03:04:41.042 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-04-06T03:04:41.102 DEBUG:teuthology.parallel:result is None 2024-04-06T03:04:48.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-06T03:04:48.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:04:48.809 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-04-06T03:04:49.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-06T03:04:49.408 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-04-06T03:04:49.408 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-04-06T03:04:49.422 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-432.el9.x86_64 2024-04-06T03:04:49.422 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T03:04:49.423 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T03:04:49.423 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T03:04:50.423 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-06T03:04:50.424 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T03:04:50.517 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-04-06T03:04:50.559 DEBUG:teuthology.parallel:result is None 2024-04-06T03:04:50.560 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T03:04:50.567 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T03:04:50.568 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T03:04:50.571 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T03:04:50.588 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T03:04:50.619 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T03:04:50.628 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T03:04:50.628 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T03:04:50.631 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T03:04:50.673 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T03:04:50.680 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T03:04:50.681 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T03:04:50.709 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T03:04:50.741 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T03:04:50.753 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T03:04:50.755 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T03:04:50.766 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T03:04:50.768 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T03:04:50.775 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T03:04:50.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T03:04:50.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T03:04:50.827 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T03:04:50.837 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T03:04:50.838 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T03:04:50.870 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T03:04:50.888 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T03:04:50.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T03:04:51.026 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T03:04:51.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T03:04:51.151 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:04:51.151 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T03:04:51.218 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T03:04:51.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T03:04:51.315 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T03:04:51.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T03:04:51.438 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:04:51.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T03:04:51.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-04-06T03:04:51.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-04-06T03:04:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T03:04:51.600 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T03:04:52.029 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T03:04:52.036 INFO:teuthology.task.internal:Starting timer... 2024-04-06T03:04:52.036 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T03:04:52.047 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T03:04:52.057 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-06T03:04:52.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-04-06T03:04:52.117 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-04-06T03:04:52.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-04-06T03:04:52.167 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-04-06T03:04:52.168 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T03:04:52.169 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-04-06T03:04:52.203 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-04-06T03:04:52.203 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-04-06T03:04:52.219 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-04-06T03:04:52.219 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-04-06T03:04:52.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T03:04:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:04:52.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T03:04:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:04:52.301 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T03:04:52.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-04-06T03:04:52.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-04-06T03:04:52.397 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T03:04:52.407 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T03:04:52.425 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T03:04:52.426 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T03:09:18.254 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-04-06T03:09:18.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:09:18.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:09:18.350 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-04-06T03:09:18.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-06T03:09:18.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-06T03:09:18.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:09:18.516 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-04-06T03:09:18.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-06T03:09:18.594 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T03:09:18.606 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T03:09:18.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T03:09:18.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:09:18.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T03:09:18.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:09:18.640 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T03:09:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T03:09:18.665 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T03:09:18.681 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T03:09:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-04-06T03:09:18.702 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-04-06T03:09:18.710 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-04-06T03:09:18.719 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T03:09:18.724 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-04-06T03:09:18.736 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T03:09:18.741 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T03:09:18.757 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T03:09:18.786 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.789 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.810 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.813 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:09:18.814 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-06T03:09:18.930 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'} 2024-04-06T03:09:18.930 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-06T03:09:18.931 INFO:tasks.cephadm:Cluster fsid is 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:09:18.931 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-06T03:09:18.931 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-06T03:09:18.931 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi060': '172.21.15.60', 'mon.smithi195': '172.21.15.195'} 2024-04-06T03:09:18.931 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-06T03:09:18.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo hostname $(hostname -s) 2024-04-06T03:09:18.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-04-06T03:09:18.986 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-06T03:09:18.987 DEBUG:teuthology.orchestra.run.smithi060:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.377 INFO:teuthology.orchestra.run.smithi060.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 6 03:09 /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.378 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.662 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 6 03:09 /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.663 DEBUG:teuthology.orchestra.run.smithi060:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.680 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:19.704 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-06T03:09:19.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-06T03:09:19.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-06T03:09:20.571 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T03:09:20.582 INFO:teuthology.orchestra.run.smithi195.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T03:09:47.737 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-06T03:09:47.737 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-06T03:09:47.737 INFO:teuthology.orchestra.run.smithi195.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-06T03:09:47.738 INFO:teuthology.orchestra.run.smithi195.stdout: "repo_digests": [ 2024-04-06T03:09:47.738 INFO:teuthology.orchestra.run.smithi195.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-06T03:09:47.738 INFO:teuthology.orchestra.run.smithi195.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-06T03:09:47.738 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-04-06T03:09:47.738 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: "repo_digests": [ 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-06T03:09:48.080 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:09:48.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph 2024-04-06T03:09:48.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-04-06T03:09:48.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /etc/ceph 2024-04-06T03:09:48.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-04-06T03:09:48.237 INFO:tasks.cephadm:Writing seed config... 2024-04-06T03:09:48.238 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-06T03:09:48.238 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-06T03:09:48.238 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-06T03:09:48.238 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-06T03:09:48.238 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-06T03:09:48.239 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-06T03:09:48.239 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-06T03:09:48.239 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-06T03:09:48.239 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:09:48.239 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-06T03:09:48.256 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-06T03:09:48.256 DEBUG:teuthology.orchestra.run.smithi060:mon.smithi060> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service 2024-04-06T03:09:48.298 INFO:tasks.cephadm:Bootstrapping... 2024-04-06T03:09:48.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.60 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:09:48.497 INFO:teuthology.orchestra.run.smithi060.stderr:-------------------------------------------------------------------------------- 2024-04-06T03:09:48.497 INFO:teuthology.orchestra.run.smithi060.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '0ee7dd52-f3c3-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.60', '--skip-admin-label'] 2024-04-06T03:09:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: 4.9.4-dev 2024-04-06T03:09:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying podman|docker is present... 2024-04-06T03:09:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: 4.9.4-dev 2024-04-06T03:09:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying lvm2 is present... 2024-04-06T03:09:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying time synchronization is in place... 2024-04-06T03:09:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T03:09:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:09:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: enabled 2024-04-06T03:09:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: active 2024-04-06T03:09:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:Unit chronyd.service is enabled and running 2024-04-06T03:09:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:Repeating the final host check... 2024-04-06T03:09:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: 4.9.4-dev 2024-04-06T03:09:48.593 INFO:teuthology.orchestra.run.smithi060.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-06T03:09:48.593 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl is present 2024-04-06T03:09:48.593 INFO:teuthology.orchestra.run.smithi060.stderr:lvcreate is present 2024-04-06T03:09:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T03:09:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:09:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: enabled 2024-04-06T03:09:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: active 2024-04-06T03:09:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:Unit chronyd.service is enabled and running 2024-04-06T03:09:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:Host looks OK 2024-04-06T03:09:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:Cluster fsid: 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:09:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:Acquiring lock 139648349671776 on /run/cephadm/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.lock 2024-04-06T03:09:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:Lock 139648349671776 acquired on /run/cephadm/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.lock 2024-04-06T03:09:48.618 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying IP 172.21.15.60 port 3300 ... 2024-04-06T03:09:48.619 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying IP 172.21.15.60 port 6789 ... 2024-04-06T03:09:48.619 INFO:teuthology.orchestra.run.smithi060.stderr:Base mon IP is 172.21.15.60, final addrv is [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2024-04-06T03:09:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.60 metric 100 2024-04-06T03:09:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.60 metric 100 2024-04-06T03:09:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-06T03:09:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-06T03:09:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-06T03:09:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:Mon IP `172.21.15.60` is in CIDR network `172.21.0.0/20` 2024-04-06T03:09:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-06T03:09:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T03:09:48.731 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-06T03:09:49.147 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Getting image source signatures 2024-04-06T03:09:49.147 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-06T03:09:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-06T03:09:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-06T03:09:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-06T03:09:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-06T03:09:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-06T03:09:49.259 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: Writing manifest to image destination 2024-04-06T03:09:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-06T03:09:49.755 INFO:teuthology.orchestra.run.smithi060.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-06T03:09:50.228 INFO:teuthology.orchestra.run.smithi060.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-06T03:09:50.228 INFO:teuthology.orchestra.run.smithi060.stderr:Extracting ceph user uid/gid from container image... 2024-04-06T03:09:50.627 INFO:teuthology.orchestra.run.smithi060.stderr:stat: 167 167 2024-04-06T03:09:51.216 INFO:teuthology.orchestra.run.smithi060.stderr:Creating initial keys... 2024-04-06T03:09:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph-authtool: AQD/vBBmBRQZHxAA4wYfYjetUaW/7Gi11OmLGQ== 2024-04-06T03:09:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph-authtool: AQAAvRBmd7pnFBAAK/iMiSDSzg04TURdH6PGkQ== 2024-04-06T03:09:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph-authtool: AQABvRBmEa28EhAAxeXhgXA1bc/VCvGXmHibXw== 2024-04-06T03:09:53.874 INFO:teuthology.orchestra.run.smithi060.stderr:Creating initial monmap... 2024-04-06T03:09:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T03:09:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-06T03:09:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:09:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T03:09:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:monmaptool for smithi060 [v2:172.21.15.60:3300,v1:172.21.15.60:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T03:09:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:setting min_mon_release = octopus 2024-04-06T03:09:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: set fsid to 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:09:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T03:09:54.772 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:09:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:Creating mon... 2024-04-06T03:09:55.702 INFO:teuthology.orchestra.run.smithi060.stderr:create mon.smithi060 on 2024-04-06T03:09:55.897 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-06T03:09:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target. 2024-04-06T03:09:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target. 2024-04-06T03:09:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Failed to reset failed state of unit ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service: Unit ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service not loaded. 2024-04-06T03:09:56.602 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target.wants/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service → /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@.service. 2024-04-06T03:09:57.233 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:57 smithi060 podman[24521]: 2024-04-06 03:09:57.197256861 +0000 UTC m=+0.305143483 container start 1859afc1bc87e213013fadd16ad2de8bd5ba1d5181d8c512002b40ee9d24980e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_BRANCH=HEAD, version=8, io.openshift.expose-services=, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T03:09:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: disabled 2024-04-06T03:09:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:09:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:firewalld.service is not enabled 2024-04-06T03:09:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:Not possible to enable service . firewalld.service is not available 2024-04-06T03:09:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mon to start... 2024-04-06T03:09:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mon... 2024-04-06T03:09:57.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:57 smithi060 ceph-mon[24540]: mkfs 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:09:57.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:57 smithi060 ceph-mon[24540]: mon.smithi060 is new leader, mons smithi060 in quorum (ranks 0) 2024-04-06T03:09:57.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:57 smithi060 bash[24521]: 1859afc1bc87e213013fadd16ad2de8bd5ba1d5181d8c512002b40ee9d24980e 2024-04-06T03:09:57.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:57 smithi060 systemd[1]: Started Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:09:58.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:58 smithi060 ceph-mon[24540]: mon.smithi060 is new leader, mons smithi060 in quorum (ranks 0) 2024-04-06T03:09:58.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:58 smithi060 ceph-mon[24540]: monmap e1: 1 mons at {smithi060=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} 2024-04-06T03:09:58.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:58 smithi060 ceph-mon[24540]: fsmap 2024-04-06T03:09:58.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:58 smithi060 ceph-mon[24540]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T03:09:58.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:09:58 smithi060 ceph-mon[24540]: mgrmap e1: no daemons active 2024-04-06T03:10:00.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:00 smithi060 ceph-mon[24540]: from='client.? 172.21.15.60:0/1897308871' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-06T03:10:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: cluster: 2024-04-06T03:10:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: id: 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: services: 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi060 (age 3s) 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: data: 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: pgs: 2024-04-06T03:10:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:06.846 INFO:teuthology.orchestra.run.smithi060.stderr:mon is available 2024-04-06T03:10:06.846 INFO:teuthology.orchestra.run.smithi060.stderr:Assimilating anything we can from ceph.conf... 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [global] 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: fsid = 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [mgr] 2024-04-06T03:10:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-06T03:10:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [osd] 2024-04-06T03:10:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-06T03:10:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-06T03:10:08.218 INFO:teuthology.orchestra.run.smithi060.stderr:Generating new minimal ceph.conf... 2024-04-06T03:10:08.469 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:08 smithi060 ceph-mon[24540]: from='client.? 172.21.15.60:0/3299523529' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T03:10:08.469 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:08 smithi060 ceph-mon[24540]: from='client.? 172.21.15.60:0/3299523529' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T03:10:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:Restarting the monitor... 2024-04-06T03:10:09.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 ceph-mon[24540]: from='client.? 172.21.15.60:0/1716806565' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:10:09.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 systemd[1]: Stopping Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:10:09.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 bash[24760]: Error: no container with name or ID "ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon.smithi060" found: no such container 2024-04-06T03:10:09.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[24536]: 2024-04-06T03:10:09.405+0000 7f5d928fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi060 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:10:09.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[24536]: 2024-04-06T03:10:09.405+0000 7f5d928fa700 -1 mon.smithi060@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T03:10:09.849 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 podman[24771]: 2024-04-06 03:10:09.610333421 +0000 UTC m=+0.287059054 container died 1859afc1bc87e213013fadd16ad2de8bd5ba1d5181d8c512002b40ee9d24980e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public, version=8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T03:10:10.170 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 podman[24771]: 2024-04-06 03:10:09.963207392 +0000 UTC m=+0.639933031 container cleanup 1859afc1bc87e213013fadd16ad2de8bd5ba1d5181d8c512002b40ee9d24980e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, version=8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T03:10:10.170 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 bash[24771]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060 2024-04-06T03:10:10.170 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:09 smithi060 bash[24801]: Error: no container with name or ID "ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon.smithi060" found: no such container 2024-04-06T03:10:10.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 podman[24785]: 2024-04-06 03:10:10.280975221 +0000 UTC m=+0.664951344 container remove 1859afc1bc87e213013fadd16ad2de8bd5ba1d5181d8c512002b40ee9d24980e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, ceph=True, release=754, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-04-06T03:10:10.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service: Deactivated successfully. 2024-04-06T03:10:10.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 systemd[1]: Stopped Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:10:10.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 systemd[1]: Starting Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:10:10.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 podman[24876]: 2024-04-06 03:10:10.420805891 +0000 UTC m=+0.018218827 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-06T03:10:10.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 podman[24876]: 2024-04-06 03:10:10.53163222 +0000 UTC m=+0.129045175 container create 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=754, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T03:10:10.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 podman[24876]: 2024-04-06 03:10:10.750768387 +0000 UTC m=+0.348181304 container init 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.tags=base centos centos-stream, release=754) 2024-04-06T03:10:10.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 podman[24876]: 2024-04-06 03:10:10.753747993 +0000 UTC m=+0.351160909 container start 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public) 2024-04-06T03:10:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-06T03:10:11.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:10:11.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: pidfile_write: ignore empty --pid-file 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: load: jerasure load: lrc 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: RocksDB version: 6.15.5 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Compile date Apr 18 2022 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: DB SUMMARY 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: DB Session ID: H7WADUC3ESQGAV4W0SA7 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: CURRENT file: CURRENT 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi060/store.db dir, Total Num: 1, files: 000008.sst 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi060/store.db: 000010.log size: 84012 ; 2024-04-06T03:10:11.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.error_if_exists: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.create_if_missing: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.paranoid_checks: 1 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.env: 0x563d040c6860 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.fs: Posix File System 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.info_log: 0x563d0543dfa0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.statistics: (nil) 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.use_fsync: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_log_file_size: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:10:11.085 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_fallocate: 1 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.use_direct_reads: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.db_log_dir: 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi060/store.db 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:10:11.086 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.write_buffer_manager: 0x563d0552e330 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:10:11.088 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.unordered_write: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.row_cache: None 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.wal_filter: None 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.preserve_deletes: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.two_write_queues: 0 2024-04-06T03:10:11.089 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.atomic_flush: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.log_readahead_size: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_background_jobs: 2 2024-04-06T03:10:11.090 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_background_compactions: -1 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_subcompactions: 1 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:10:11.091 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_open_files: -1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_background_flushes: -1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Compression algorithms supported: 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kZSTD supported: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kXpressCompression supported: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kLZ4Compression supported: 1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kBZip2Compression supported: 0 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kZlibCompression supported: 1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: kSnappyCompression supported: 1 2024-04-06T03:10:11.092 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi060/store.db/MANIFEST-000009 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.merge_operator: 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_filter: None 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d05409d00) 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: cache_index_and_filter_blocks: 1 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: pin_top_level_index_and_filter: 1 2024-04-06T03:10:11.093 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_type: 0 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: data_block_index_type: 0 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_shortening: 1 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: hash_index_allow_collision: 1 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: checksum: 1 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: no_block_cache: 0 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache: 0x563d05474170 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_name: BinnedLRUCache 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_options: 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: capacity : 536870912 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: num_shard_bits : 4 2024-04-06T03:10:11.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: strict_capacity_limit : 0 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: high_pri_pool_ratio: 0.000 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_compressed: (nil) 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: persistent_cache: (nil) 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_size: 4096 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_size_deviation: 10 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_restart_interval: 16 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_block_restart_interval: 1 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: metadata_block_size: 4096 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: partition_filters: 0 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: use_delta_encoding: 1 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-06T03:10:11.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: whole_key_filtering: 1 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: verify_compression: 0 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: read_amp_bytes_per_bit: 0 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: format_version: 4 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: enable_index_compression: 1 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_align: 0 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression: NoCompression 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.num_levels: 7 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:10:11.096 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:11.097 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:10:11.098 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.arena_block_size: 4194304 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:10:11.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.table_properties_collectors: 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.inplace_update_support: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.bloom_locality: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.max_successive_merges: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:10:11.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.ttl: 2592000 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.enable_blob_files: false 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.min_blob_size: 0 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi060/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373010779449, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-06T03:10:11.101 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373010780153, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80060, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78323, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9946, "raw_average_key_size": 46, "raw_value_size": 72579, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712373010, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "dee4c115-09e3-4c0f-8a3f-8497cdfe4bd3", "db_session_id": "H7WADUC3ESQGAV4W0SA7"}} 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373010780480, "job": 1, "event": "recovery_finished"} 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi060/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563d0545aa80 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: DB pointer 0x563d0546a000 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** DB Stats ** 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:11.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: L0 2/0 79.98 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Sum 2/0 79.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:11.103 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative compaction: 0.00 GB write, 25.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval compaction: 0.00 GB write, 25.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T03:10:11.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: L0 2/0 79.98 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Sum 2/0 79.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.4 0.00 0.00 1 0.000 0 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:10:11.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative compaction: 0.00 GB write, 25.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: starting mon.smithi060 rank 0 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi060 fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???) e1 preinit fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).mds e1 new map 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).mds e1 print_map 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: e1 2024-04-06T03:10:11.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: legacy client fscid: -1 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: No filesystems configured 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:11.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expand map: {default=false} 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta from 'false' to 'false' 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expanded map: {default=false} 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expand map: {default=info} 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta from 'info' to 'info' 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expanded map: {default=info} 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expand map: {default=daemon} 2024-04-06T03:10:11.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expanded map: {default=daemon} 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expand map: {default=debug} 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta from 'debug' to 'debug' 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: expand_channel_meta expanded map: {default=debug} 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mon.smithi060 is new leader, mons smithi060 in quorum (ranks 0) 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: monmap e1: 1 mons at {smithi060=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: fsmap 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 ceph-mon[24896]: mgrmap e1: no daemons active 2024-04-06T03:10:11.109 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 bash[24876]: 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd 2024-04-06T03:10:11.110 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:10 smithi060 systemd[1]: Started Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:10:11.844 INFO:teuthology.orchestra.run.smithi060.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-06T03:10:11.844 INFO:teuthology.orchestra.run.smithi060.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:10:11.844 INFO:teuthology.orchestra.run.smithi060.stderr:Creating mgr... 2024-04-06T03:10:11.845 INFO:teuthology.orchestra.run.smithi060.stderr:Verifying port 9283 ... 2024-04-06T03:10:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Failed to reset failed state of unit ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mgr.smithi060.jttwce.service: Unit ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mgr.smithi060.jttwce.service not loaded. 2024-04-06T03:10:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4.target.wants/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mgr.smithi060.jttwce.service → /etc/systemd/system/ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@.service. 2024-04-06T03:10:12.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:12 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1563863164' entity='client.admin' 2024-04-06T03:10:15.030 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: disabled 2024-04-06T03:10:15.038 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:firewalld.service is not enabled 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:Not possible to enable service . firewalld.service is not available 2024-04-06T03:10:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: disabled 2024-04-06T03:10:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:10:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:firewalld.service is not enabled 2024-04-06T03:10:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-06T03:10:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mgr to start... 2024-04-06T03:10:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mgr... 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsid": "0ee7dd52-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "health": { 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T03:10:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 0 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "smithi060" 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T03:10:15.578 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:15.579 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T03:10:15.579 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T03:10:15.579 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T03:10:15.579 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T03:10:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:15.584 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": false, 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "iostat", 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "nfs", 2024-04-06T03:10:15.585 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "restful" 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modified": "2024-04-06T03:09:57.256375+0000", 2024-04-06T03:10:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:15.587 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:15.587 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T03:10:15.587 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:15.733 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:15 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1165450637' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:mgr not available, waiting (1/15)... 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsid": "0ee7dd52-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "health": { 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T03:10:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 0 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "smithi060" 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T03:10:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T03:10:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": false, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T03:10:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "iostat", 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "nfs", 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "restful" 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modified": "2024-04-06T03:09:57.256375+0000", 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T03:10:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:18.965 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:18 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2989514776' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:mgr not available, waiting (2/15)... 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsid": "0ee7dd52-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "health": { 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 0 2024-04-06T03:10:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "smithi060" 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T03:10:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": false, 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T03:10:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "iostat", 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "nfs", 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "restful" 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T03:10:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modified": "2024-04-06T03:09:57.256375+0000", 2024-04-06T03:10:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T03:10:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: Activating manager daemon smithi060.jttwce 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: mgrmap e2: smithi060.jttwce(active, starting, since 0.00282536s) 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:10:21.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='mgr.14100 172.21.15.60:0/585954836' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:21.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:21 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/36140255' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:22.305 INFO:teuthology.orchestra.run.smithi060.stderr:mgr not available, waiting (3/15)... 2024-04-06T03:10:23.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:22 smithi060 ceph-mon[24896]: mgrmap e3: smithi060.jttwce(active, since 1.00561s) 2024-04-06T03:10:24.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:24 smithi060 ceph-mon[24896]: mgrmap e4: smithi060.jttwce(active, since 3s) 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsid": "0ee7dd52-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "health": { 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T03:10:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 0 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "smithi060" 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T03:10:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T03:10:24.985 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T03:10:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T03:10:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T03:10:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": true, 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "iostat", 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "nfs", 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "restful" 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ], 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "modified": "2024-04-06T03:09:57.256375+0000", 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "services": {} 2024-04-06T03:10:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: }, 2024-04-06T03:10:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T03:10:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:25.568 INFO:teuthology.orchestra.run.smithi060.stderr:mgr is available 2024-04-06T03:10:25.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:25 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2786359444' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [global] 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: fsid = 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-06T03:10:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [mgr] 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: [osd] 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-06T03:10:26.206 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-06T03:10:26.775 INFO:teuthology.orchestra.run.smithi060.stderr:Enabling cephadm module... 2024-04-06T03:10:27.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:26 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3797121744' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T03:10:28.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:27 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/703574466' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-06T03:10:29.164 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:28 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/703574466' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-06T03:10:29.165 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:28 smithi060 ceph-mon[24896]: mgrmap e5: smithi060.jttwce(active, since 7s) 2024-04-06T03:10:29.197 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:29.197 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-06T03:10:29.197 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": true, 2024-04-06T03:10:29.197 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "active_name": "smithi060.jttwce", 2024-04-06T03:10:29.198 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-06T03:10:29.198 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:29.814 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for the mgr to restart... 2024-04-06T03:10:29.814 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mgr epoch 5... 2024-04-06T03:10:30.029 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:29 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3872540679' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T03:10:35.006 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:35.006 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-06T03:10:35.007 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "initialized": true 2024-04-06T03:10:35.007 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: Active manager daemon smithi060.jttwce restarted 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: Activating manager daemon smithi060.jttwce 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: osdmap e2: 0 total, 0 up, 0 in 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: mgrmap e6: smithi060.jttwce(active, starting, since 0.0526389s) 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:10:35.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:10:35.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:34 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:10:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:mgr epoch 5 is available 2024-04-06T03:10:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:Setting orchestrator backend to cephadm... 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: mgrmap e7: smithi060.jttwce(active, since 1.0548s) 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: [06/Apr/2024:03:10:35] ENGINE Bus STARTING 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: [06/Apr/2024:03:10:35] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: [06/Apr/2024:03:10:35] ENGINE Bus STARTED 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:36.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:36 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:37.457 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: value unchanged 2024-04-06T03:10:37.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:37 smithi060 ceph-mon[24896]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:37.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:37 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:37.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:37 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:37.735 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:37 smithi060 ceph-mon[24896]: mgrmap e8: smithi060.jttwce(active, since 2s) 2024-04-06T03:10:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:Generating ssh key... 2024-04-06T03:10:38.329 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:38 smithi060 ceph-mon[24896]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:39.820 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:39 smithi060 ceph-mon[24896]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:39.820 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:39 smithi060 ceph-mon[24896]: Generating ssh key... 2024-04-06T03:10:39.820 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:39 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:39.821 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:39 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYy0yZb4BHrzgPuUKoPziJTajBfFQ645tE0SsD/sPAGkGFP4NQmCCnxv79AjUGmbi1xj1ih6sJKtpdVaBQHcJBWDXfVyv1YNPbm/H2oM295mXTK6vAG3d8pfjoQjUPI2yFlTMmDLUyCXKnDxaQMR9Q82sKdzalymbRMNUH/sM4zO7dnUJAbroOAAxm6+BZguiHSpi3sAMyaVMdKZMjeb++FbZ8fhhp878qs8YN+thd3TeVymzXSqJzCGZUhjp9cc7tNuRUk0UVMk1ppO4QaZIFSadEs1ooA9KNgKZ4nOh52/MwVbsvB13BHPEI6bZ9cj/ZNb4/pNspiiUb7WSvpJE/LatmcdmEepFSENMCG+pd+UJA7Sdc7ccGFbPNfAGgZHw8g0htW+xCFpIjmMpUnr8iUEK8fmzLBf4Yo7rekKXo3L0VAGi5EGNzH94uB+v05QpWFx155VBlf2IWjb+AzGFg7alf4x2lCoqLLaVDM9SjpbQBYI/yDSM+ijVYI0MNlGk= ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:10:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:Adding key to root@localhost authorized_keys... 2024-04-06T03:10:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:Adding host smithi060... 2024-04-06T03:10:40.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:40 smithi060 ceph-mon[24896]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Added host 'smithi060' with addr '172.21.15.60' 2024-04-06T03:10:41.984 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:41 smithi060 ceph-mon[24896]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi060", "addr": "172.21.15.60", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:42.582 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying mon service with default placement... 2024-04-06T03:10:43.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:42 smithi060 ceph-mon[24896]: Deploying cephadm binary to smithi060 2024-04-06T03:10:43.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:42 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:43.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:42 smithi060 ceph-mon[24896]: Added host smithi060 2024-04-06T03:10:43.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:42 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-06T03:10:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying mgr service with default placement... 2024-04-06T03:10:44.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:44 smithi060 ceph-mon[24896]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:44.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:44 smithi060 ceph-mon[24896]: Saving service mon spec with placement count:5 2024-04-06T03:10:44.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:44 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:44.781 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-06T03:10:45.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:45 smithi060 ceph-mon[24896]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:45.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:45 smithi060 ceph-mon[24896]: Saving service mgr spec with placement count:2 2024-04-06T03:10:45.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:45 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:45.897 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:45 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:46.278 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying crash service with default placement... 2024-04-06T03:10:46.894 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-06T03:10:47.168 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:46 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:47.169 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:46 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:48.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:47 smithi060 ceph-mon[24896]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:48.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:47 smithi060 ceph-mon[24896]: Saving service crash spec with placement * 2024-04-06T03:10:48.102 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:47 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying prometheus service with default placement... 2024-04-06T03:10:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-06T03:10:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying grafana service with default placement... 2024-04-06T03:10:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-06T03:10:49.935 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:49 smithi060 ceph-mon[24896]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:49.935 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:49 smithi060 ceph-mon[24896]: Saving service prometheus spec with placement count:1 2024-04-06T03:10:49.935 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:49 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:50.492 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying node-exporter service with default placement... 2024-04-06T03:10:50.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:50 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:50.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:50 smithi060 ceph-mon[24896]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:50.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:50 smithi060 ceph-mon[24896]: Saving service grafana spec with placement count:1 2024-04-06T03:10:50.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:50 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-06T03:10:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:Deploying alertmanager service with default placement... 2024-04-06T03:10:52.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:52 smithi060 ceph-mon[24896]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:52.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:52 smithi060 ceph-mon[24896]: Saving service node-exporter spec with placement * 2024-04-06T03:10:52.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:52 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-06T03:10:54.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:54 smithi060 ceph-mon[24896]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:10:54.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:54 smithi060 ceph-mon[24896]: Saving service alertmanager spec with placement count:1 2024-04-06T03:10:54.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:54 smithi060 ceph-mon[24896]: from='mgr.14120 172.21.15.60:0/427425900' entity='mgr.smithi060.jttwce' 2024-04-06T03:10:55.375 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:55 smithi060 ceph-mon[24896]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:10:55.375 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:55 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1370533349' entity='client.admin' 2024-04-06T03:10:56.188 INFO:teuthology.orchestra.run.smithi060.stderr:Enabling the dashboard module... 2024-04-06T03:10:56.985 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:56 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3995195966' entity='client.admin' 2024-04-06T03:10:58.111 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:57 smithi060 ceph-mon[24896]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:10:58.111 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:57 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3835320102' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-06T03:10:58.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:58 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3835320102' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-06T03:10:58.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:58 smithi060 ceph-mon[24896]: mgrmap e9: smithi060.jttwce(active, since 23s) 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "available": true, 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "active_name": "smithi060.jttwce", 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-06T03:10:58.976 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:10:59.536 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for the mgr to restart... 2024-04-06T03:10:59.536 INFO:teuthology.orchestra.run.smithi060.stderr:Waiting for mgr epoch 9... 2024-04-06T03:10:59.885 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:10:59 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2251207782' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T03:11:04.823 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: { 2024-04-06T03:11:04.823 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-06T03:11:04.823 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: "initialized": true 2024-04-06T03:11:04.823 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: } 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: Active manager daemon smithi060.jttwce restarted 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: Activating manager daemon smithi060.jttwce 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: osdmap e3: 0 total, 0 up, 0 in 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: mgrmap e10: smithi060.jttwce(active, starting, since 0.0525517s) 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:11:05.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:11:05.063 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:05.063 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:05.063 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:11:05.063 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:04 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:11:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:mgr epoch 9 is available 2024-04-06T03:11:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:Generating a dashboard self-signed certificate... 2024-04-06T03:11:05.879 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:05 smithi060 ceph-mon[24896]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T03:11:05.879 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:05 smithi060 ceph-mon[24896]: mgrmap e11: smithi060.jttwce(active, since 1.05539s) 2024-04-06T03:11:05.879 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:05 smithi060 ceph-mon[24896]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T03:11:05.879 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:05 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-06T03:11:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:Creating initial admin user... 2024-04-06T03:11:07.333 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:07.333 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:07.333 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:07.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:06] ENGINE Bus STARTING 2024-04-06T03:11:07.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:06] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:11:07.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:06] ENGINE Bus STARTED 2024-04-06T03:11:07.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:07.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:07 smithi060 ceph-mon[24896]: mgrmap e12: smithi060.jttwce(active, since 2s) 2024-04-06T03:11:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$cqWu9KvByM4tWY82cYrRtuYuGUq3Dwtw89puo.LcYYlu3wDI2wyZm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712373068, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-06T03:11:08.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:08 smithi060 ceph-mon[24896]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:08.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:08 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:Fetching dashboard port number... 2024-04-06T03:11:09.451 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: 8443 2024-04-06T03:11:09.735 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:09 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/525766265' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-06T03:11:10.317 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: disabled 2024-04-06T03:11:10.325 INFO:teuthology.orchestra.run.smithi060.stderr:systemctl: inactive 2024-04-06T03:11:10.326 INFO:teuthology.orchestra.run.smithi060.stderr:firewalld.service is not enabled 2024-04-06T03:11:10.326 INFO:teuthology.orchestra.run.smithi060.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-06T03:11:10.328 INFO:teuthology.orchestra.run.smithi060.stderr:Ceph Dashboard is now available at: 2024-04-06T03:11:10.328 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:10.329 INFO:teuthology.orchestra.run.smithi060.stderr: URL: https://smithi060.front.sepia.ceph.com:8443/ 2024-04-06T03:11:10.329 INFO:teuthology.orchestra.run.smithi060.stderr: User: admin 2024-04-06T03:11:10.329 INFO:teuthology.orchestra.run.smithi060.stderr: Password: 2ptb5ktzl4 2024-04-06T03:11:10.329 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:10.329 INFO:teuthology.orchestra.run.smithi060.stderr:Enabling autotune for osd_memory_target 2024-04-06T03:11:11.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:11 smithi060 ceph-mon[24896]: mgrmap e13: smithi060.jttwce(active, since 6s) 2024-04-06T03:11:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr:You can access the Ceph CLI with: 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: ceph telemetry on 2024-04-06T03:11:12.912 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:For more information see: 2024-04-06T03:11:12.913 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.913 INFO:teuthology.orchestra.run.smithi060.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-06T03:11:12.913 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:11:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:Bootstrap complete. 2024-04-06T03:11:12.949 INFO:tasks.cephadm:Fetching config... 2024-04-06T03:11:12.949 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:11:12.949 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-06T03:11:12.966 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-06T03:11:12.966 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:11:12.966 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-06T03:11:13.020 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-06T03:11:13.021 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:11:13.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/keyring of=/dev/stdout 2024-04-06T03:11:13.087 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-06T03:11:13.087 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:11:13.087 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-06T03:11:13.142 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-06T03:11:13.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYy0yZb4BHrzgPuUKoPziJTajBfFQ645tE0SsD/sPAGkGFP4NQmCCnxv79AjUGmbi1xj1ih6sJKtpdVaBQHcJBWDXfVyv1YNPbm/H2oM295mXTK6vAG3d8pfjoQjUPI2yFlTMmDLUyCXKnDxaQMR9Q82sKdzalymbRMNUH/sM4zO7dnUJAbroOAAxm6+BZguiHSpi3sAMyaVMdKZMjeb++FbZ8fhhp878qs8YN+thd3TeVymzXSqJzCGZUhjp9cc7tNuRUk0UVMk1ppO4QaZIFSadEs1ooA9KNgKZ4nOh52/MwVbsvB13BHPEI6bZ9cj/ZNb4/pNspiiUb7WSvpJE/LatmcdmEepFSENMCG+pd+UJA7Sdc7ccGFbPNfAGgZHw8g0htW+xCFpIjmMpUnr8iUEK8fmzLBf4Yo7rekKXo3L0VAGi5EGNzH94uB+v05QpWFx155VBlf2IWjb+AzGFg7alf4x2lCoqLLaVDM9SjpbQBYI/yDSM+ijVYI0MNlGk= ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T03:11:13.224 INFO:teuthology.orchestra.run.smithi060.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYy0yZb4BHrzgPuUKoPziJTajBfFQ645tE0SsD/sPAGkGFP4NQmCCnxv79AjUGmbi1xj1ih6sJKtpdVaBQHcJBWDXfVyv1YNPbm/H2oM295mXTK6vAG3d8pfjoQjUPI2yFlTMmDLUyCXKnDxaQMR9Q82sKdzalymbRMNUH/sM4zO7dnUJAbroOAAxm6+BZguiHSpi3sAMyaVMdKZMjeb++FbZ8fhhp878qs8YN+thd3TeVymzXSqJzCGZUhjp9cc7tNuRUk0UVMk1ppO4QaZIFSadEs1ooA9KNgKZ4nOh52/MwVbsvB13BHPEI6bZ9cj/ZNb4/pNspiiUb7WSvpJE/LatmcdmEepFSENMCG+pd+UJA7Sdc7ccGFbPNfAGgZHw8g0htW+xCFpIjmMpUnr8iUEK8fmzLBf4Yo7rekKXo3L0VAGi5EGNzH94uB+v05QpWFx155VBlf2IWjb+AzGFg7alf4x2lCoqLLaVDM9SjpbQBYI/yDSM+ijVYI0MNlGk= ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:13.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYy0yZb4BHrzgPuUKoPziJTajBfFQ645tE0SsD/sPAGkGFP4NQmCCnxv79AjUGmbi1xj1ih6sJKtpdVaBQHcJBWDXfVyv1YNPbm/H2oM295mXTK6vAG3d8pfjoQjUPI2yFlTMmDLUyCXKnDxaQMR9Q82sKdzalymbRMNUH/sM4zO7dnUJAbroOAAxm6+BZguiHSpi3sAMyaVMdKZMjeb++FbZ8fhhp878qs8YN+thd3TeVymzXSqJzCGZUhjp9cc7tNuRUk0UVMk1ppO4QaZIFSadEs1ooA9KNgKZ4nOh52/MwVbsvB13BHPEI6bZ9cj/ZNb4/pNspiiUb7WSvpJE/LatmcdmEepFSENMCG+pd+UJA7Sdc7ccGFbPNfAGgZHw8g0htW+xCFpIjmMpUnr8iUEK8fmzLBf4Yo7rekKXo3L0VAGi5EGNzH94uB+v05QpWFx155VBlf2IWjb+AzGFg7alf4x2lCoqLLaVDM9SjpbQBYI/yDSM+ijVYI0MNlGk= ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T03:11:13.292 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYy0yZb4BHrzgPuUKoPziJTajBfFQ645tE0SsD/sPAGkGFP4NQmCCnxv79AjUGmbi1xj1ih6sJKtpdVaBQHcJBWDXfVyv1YNPbm/H2oM295mXTK6vAG3d8pfjoQjUPI2yFlTMmDLUyCXKnDxaQMR9Q82sKdzalymbRMNUH/sM4zO7dnUJAbroOAAxm6+BZguiHSpi3sAMyaVMdKZMjeb++FbZ8fhhp878qs8YN+thd3TeVymzXSqJzCGZUhjp9cc7tNuRUk0UVMk1ppO4QaZIFSadEs1ooA9KNgKZ4nOh52/MwVbsvB13BHPEI6bZ9cj/ZNb4/pNspiiUb7WSvpJE/LatmcdmEepFSENMCG+pd+UJA7Sdc7ccGFbPNfAGgZHw8g0htW+xCFpIjmMpUnr8iUEK8fmzLBf4Yo7rekKXo3L0VAGi5EGNzH94uB+v05QpWFx155VBlf2IWjb+AzGFg7alf4x2lCoqLLaVDM9SjpbQBYI/yDSM+ijVYI0MNlGk= ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:13.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-06T03:11:13.615 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:13 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/4163248313' entity='client.admin' 2024-04-06T03:11:16.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:15 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:16.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:15 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:11:16.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:15 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:16.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:15 smithi060 ceph-mon[24896]: Deploying daemon alertmanager.smithi060 on smithi060 2024-04-06T03:11:16.690 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-06T03:11:16.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-06T03:11:17.362 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:17 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3737904036' entity='client.admin' 2024-04-06T03:11:19.989 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-04-06T03:11:19.989 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:11:19.989 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-04-06T03:11:20.007 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:11:20.008 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:11:20.065 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-04-06T03:11:20.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi195 2024-04-06T03:11:20.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:20 smithi060 ceph-mon[24896]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:20.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:20 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:22 smithi060 ceph-mon[24896]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:23.520 INFO:teuthology.orchestra.run.smithi060.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-04-06T03:11:24.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-06T03:11:24.771 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:24 smithi060 ceph-mon[24896]: Deploying cephadm binary to smithi195 2024-04-06T03:11:24.771 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:24 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:24.771 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:24 smithi060 ceph-mon[24896]: Added host smithi195 2024-04-06T03:11:24.771 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:24 smithi060 ceph-mon[24896]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:27.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:26 smithi060 ceph-mon[24896]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:28.149 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:27 smithi060 ceph-mon[24896]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:29.470 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:11:29.470 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-04-06T03:11:30.143 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:29 smithi060 ceph-mon[24896]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:11:30.144 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:29 smithi060 ceph-mon[24896]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:31.060 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-06T03:11:31.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-06T03:11:31.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:31 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:31.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:31 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:11:31.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:31 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T03:11:31.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:31 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:11:31.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:31 smithi060 ceph-mon[24896]: Deploying daemon crash.smithi060 on smithi060 2024-04-06T03:11:32.468 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:32 smithi060 ceph-mon[24896]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:34.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:33 smithi060 ceph-mon[24896]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:34.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:33 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3558678588' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:11:35.197 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:11:35.198 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:35.198 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:34 smithi060 ceph-mon[24896]: Deploying daemon grafana.smithi060 on smithi060 2024-04-06T03:11:35.523 INFO:tasks.cephadm:Adding mon.smithi060 on smithi060 2024-04-06T03:11:35.523 INFO:tasks.cephadm:Adding mon.smithi195 on smithi195 2024-04-06T03:11:35.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi060:172.21.15.60=smithi060;smithi195:172.21.15.195=smithi195' 2024-04-06T03:11:36.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:35 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3558678588' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T03:11:36.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:35 smithi060 ceph-mon[24896]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:11:36.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:35 smithi060 ceph-mon[24896]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:37.269 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-04-06T03:11:37.771 DEBUG:teuthology.orchestra.run.smithi195:mon.smithi195> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service 2024-04-06T03:11:37.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:37.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:38.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:38 smithi060 ceph-mon[24896]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi060:172.21.15.60=smithi060;smithi195:172.21.15.195=smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:38.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:38 smithi060 ceph-mon[24896]: Saving service mon spec with placement smithi060:172.21.15.60=smithi060;smithi195:172.21.15.195=smithi195;count:2 2024-04-06T03:11:38.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:38 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:38.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:38 smithi060 ceph-mon[24896]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:39.522 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:39.524 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:39 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:39 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/1794157593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:39 smithi060 ceph-mon[24896]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:41.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:41.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:42.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:41 smithi060 ceph-mon[24896]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:42.838 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:42.839 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:42.841 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:43.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:42 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/2579431665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:44.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:43 smithi060 ceph-mon[24896]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:44.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:44.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:46.304 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:46.304 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:46.305 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:46.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:46 smithi060 ceph-mon[24896]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:46.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:46 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3063433952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:47.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:47.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:48.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:48 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:48.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:48 smithi060 ceph-mon[24896]: Deploying daemon node-exporter.smithi060 on smithi060 2024-04-06T03:11:48.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:48 smithi060 ceph-mon[24896]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:49.840 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:49.840 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:49.841 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:50.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:49 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:50.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:49 smithi060 ceph-mon[24896]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:50.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:49 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3635859038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:51.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:51.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:52.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:51 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:52.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:51 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T03:11:53.036 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:52 smithi060 ceph-mon[24896]: from='mgr.14162 172.21.15.60:0/314872557' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T03:11:53.036 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:52 smithi060 ceph-mon[24896]: mgrmap e14: smithi060.jttwce(active, since 47s) 2024-04-06T03:11:53.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:53.145 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:53.147 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:53.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:53 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/383451841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:54.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:54.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:56.408 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:56.410 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:11:57.156 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:56 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3631020365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:11:57.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:11:57.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:11:57.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:57 smithi060 ceph-mon[24896]: Active manager daemon smithi060.jttwce restarted 2024-04-06T03:11:57.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:57 smithi060 ceph-mon[24896]: Activating manager daemon smithi060.jttwce 2024-04-06T03:11:57.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:57 smithi060 ceph-mon[24896]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: mgrmap e15: smithi060.jttwce(active, starting, since 0.0714604s) 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:59.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:58] ENGINE Bus STARTING 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:58] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: [06/Apr/2024:03:11:58] ENGINE Bus STARTED 2024-04-06T03:11:59.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:11:59.723 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:11:59.723 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:11:59.725 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:12:00.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:59 smithi060 ceph-mon[24896]: mgrmap e16: smithi060.jttwce(active, since 1.07702s) 2024-04-06T03:12:00.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:00.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:00.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:11:59 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/1402970742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:01.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:12:01.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:01.591 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:01 smithi060 ceph-mon[24896]: mgrmap e17: smithi060.jttwce(active, since 2s) 2024-04-06T03:12:01.591 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:01.591 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:03.135 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:03.135 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:12:03.135 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:02 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:12:03.135 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:02 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:03.135 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:04.021 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:12:04.021 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:12:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:12:04.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:04 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/1364487675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:05.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:12:05.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:06.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:06.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:05 smithi060 ceph-mon[24896]: Deploying daemon crash.smithi195 on smithi195 2024-04-06T03:12:07.797 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:12:07.797 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:12:07.798 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:12:08.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:07 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/2401181265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:09.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:12:09.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:10.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:10 smithi060 ceph-mon[24896]: Deploying daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:12:12.238 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:12:12.238 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:12:12.240 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:12:12.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:12 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3906930673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:13.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:12:13.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:14.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:14.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:14.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:14.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:14.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:13 smithi060 ceph-mon[24896]: Deploying daemon mon.smithi195 on smithi195 2024-04-06T03:12:17.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:12:17.315 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:09:54.234976Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:12:17.316 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-06T03:12:17.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:17 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/144145756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:19.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T03:12:19.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:22.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: mon.smithi060 calling monitor election 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3877229548' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:12:22.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: mon.smithi195 calling monitor election 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: mon.smithi060 is new leader, mons smithi060,smithi195 in quorum (ranks 0,1) 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: monmap e2: 2 mons at {smithi060=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],smithi195=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} 2024-04-06T03:12:22.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: fsmap 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: mgrmap e17: smithi060.jttwce(active, since 24s) 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3877229548' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: overall HEALTH_OK 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3877229548' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:12:22.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:22.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3877229548' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:12:22.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:24.173 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:23 smithi060 ceph-mon[24896]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:24.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:23 smithi060 ceph-mon[24896]: Deploying daemon node-exporter.smithi195 on smithi195 2024-04-06T03:12:24.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:23 smithi060 ceph-mon[24896]: mgrmap e18: smithi060.jttwce(active, since 25s), standbys: smithi195.ultkot 2024-04-06T03:12:24.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:12:24.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:12:24.277 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-06T03:12:24.277 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:12:17.659725Z","created":"2024-04-06T03:09:54.234976Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi060","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi195","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T03:12:24.279 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-04-06T03:12:25.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:24 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/2207560874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:25.404 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-06T03:12:25.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-06T03:12:25.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:25 smithi060 ceph-mon[24896]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:27.297 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:27 smithi060 ceph-mon[24896]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:27.297 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:27.297 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:27 smithi060 ceph-mon[24896]: Deploying daemon prometheus.smithi060 on smithi060 2024-04-06T03:12:27.461 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:27 smithi195 ceph-mon[27147]: Deploying daemon prometheus.smithi060 on smithi060 2024-04-06T03:12:27.534 INFO:teuthology.orchestra.run.smithi060.stdout:# minimal ceph.conf for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:27.535 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-04-06T03:12:27.535 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:27.535 INFO:teuthology.orchestra.run.smithi060.stdout: mon_host = [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2024-04-06T03:12:28.253 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-06T03:12:28.253 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:12:28.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T03:12:28.282 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:12:28.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:28.349 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:12:28.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T03:12:28.384 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:12:28.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:28.450 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:28 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1083194543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:28.451 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:28.454 INFO:tasks.cephadm:Deploying OSDs... 2024-04-06T03:12:28.455 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:12:28.455 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T03:12:28.469 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T03:12:28.470 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-04-06T03:12:28.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:28 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1083194543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:28.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:28.500 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T03:12:28.500 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:28.500 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-06T03:12:28.500 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:28.500 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:28.501 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-06 03:11:48.351558169 +0000 2024-04-06T03:12:28.501 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-06 03:08:53.399256712 +0000 2024-04-06T03:12:28.501 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-06 03:08:53.399256712 +0000 2024-04-06T03:12:28.501 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-06 03:08:53.399256712 +0000 2024-04-06T03:12:28.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T03:12:28.566 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-06T03:12:28.566 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-06T03:12:28.566 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000102322 s, 5.0 MB/s 2024-04-06T03:12:28.567 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T03:12:28.623 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-04-06T03:12:28.677 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-06 03:11:48.351558169 +0000 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-06 03:08:53.736253070 +0000 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-06 03:08:53.736253070 +0000 2024-04-06T03:12:28.678 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-06 03:08:53.736253070 +0000 2024-04-06T03:12:28.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T03:12:28.743 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-06T03:12:28.743 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-06T03:12:28.743 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.7919e-05 s, 5.2 MB/s 2024-04-06T03:12:28.744 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T03:12:28.799 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-06 03:11:48.352558158 +0000 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-06 03:08:54.061249557 +0000 2024-04-06T03:12:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-06 03:08:54.061249557 +0000 2024-04-06T03:12:28.855 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-06 03:08:54.061249557 +0000 2024-04-06T03:12:28.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T03:12:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-06T03:12:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-06T03:12:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.4713e-05 s, 5.4 MB/s 2024-04-06T03:12:28.919 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T03:12:28.975 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-06 03:11:48.352558158 +0000 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-06 03:08:54.401245882 +0000 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-06 03:08:54.401245882 +0000 2024-04-06T03:12:29.029 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-06 03:08:54.401245882 +0000 2024-04-06T03:12:29.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T03:12:29.093 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-06T03:12:29.093 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-06T03:12:29.094 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000100853 s, 5.1 MB/s 2024-04-06T03:12:29.095 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T03:12:29.150 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-06T03:12:29.150 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T03:12:29.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T03:12:29.169 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-04-06T03:12:29.227 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-06 03:12:23.299981018 +0000 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-06 03:08:55.984929468 +0000 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-06 03:08:55.984929468 +0000 2024-04-06T03:12:29.228 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-06 03:08:55.984929468 +0000 2024-04-06T03:12:29.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T03:12:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-06T03:12:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-06T03:12:29.298 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000130065 s, 3.9 MB/s 2024-04-06T03:12:29.299 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T03:12:29.358 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:29.414 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-06 03:12:23.299981018 +0000 2024-04-06T03:12:29.415 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-06 03:08:56.321923084 +0000 2024-04-06T03:12:29.415 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-06 03:08:56.321923084 +0000 2024-04-06T03:12:29.415 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-06 03:08:56.321923084 +0000 2024-04-06T03:12:29.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T03:12:29.483 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-06T03:12:29.484 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-06T03:12:29.484 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000105243 s, 4.9 MB/s 2024-04-06T03:12:29.485 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T03:12:29.542 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-06 03:12:23.299981018 +0000 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-06 03:08:56.649916870 +0000 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-06 03:08:56.649916870 +0000 2024-04-06T03:12:29.599 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-06 03:08:56.649916870 +0000 2024-04-06T03:12:29.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T03:12:29.665 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-06T03:12:29.666 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-06T03:12:29.666 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.3862e-05 s, 5.5 MB/s 2024-04-06T03:12:29.667 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T03:12:29.723 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T03:12:29.779 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-06 03:12:23.300980999 +0000 2024-04-06T03:12:29.780 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-06 03:08:56.964910903 +0000 2024-04-06T03:12:29.780 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-06 03:08:56.964910903 +0000 2024-04-06T03:12:29.780 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-06 03:08:56.964910903 +0000 2024-04-06T03:12:29.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T03:12:29.846 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-06T03:12:29.846 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-06T03:12:29.846 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.8981e-05 s, 5.8 MB/s 2024-04-06T03:12:29.847 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T03:12:29.904 INFO:tasks.cephadm:Deploying osd.0 on smithi060 with /dev/vg_nvme/lv_4... 2024-04-06T03:12:29.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T03:12:29.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:29 smithi195 ceph-mon[27147]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:29.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:29 smithi060 ceph-mon[24896]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:31.878 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:31 smithi060 ceph-mon[24896]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:31.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:31 smithi195 ceph-mon[27147]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:32.836 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T03:12:32.837 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T03:12:32.837 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-04-06T03:12:32.837 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-04-06T03:12:32.837 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00848226 s, 1.2 GB/s 2024-04-06T03:12:32.837 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-04-06T03:12:33.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:33 smithi195 ceph-mon[27147]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:33.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:33 smithi060 ceph-mon[24896]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:34.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi060:vg_nvme/lv_4 2024-04-06T03:12:35.901 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:35 smithi060 ceph-mon[24896]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:35.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:35 smithi195 ceph-mon[27147]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:37.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:36 smithi195 ceph-mon[27147]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:37.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:36 smithi060 ceph-mon[24896]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:38.078 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:37 smithi060 ceph-mon[24896]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:38.078 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:37 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:12:38.078 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:37 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:12:38.078 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:37 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:38.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:37 smithi195 ceph-mon[27147]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:38.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:37 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:12:38.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:37 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:12:38.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:37 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:39.025 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:38 smithi060 ceph-mon[24896]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:39.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:38 smithi195 ceph-mon[27147]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:41.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:41 smithi060 ceph-mon[24896]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:41.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:41 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:41.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:41 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:41.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:41 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:41.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:41 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:41.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:41 smithi195 ceph-mon[27147]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:41.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:41 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:41.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:41 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:41.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:41 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:41.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:41 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:42.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2189768185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77456af4-610d-4353-8774-41203640a91d"}]: dispatch 2024-04-06T03:12:42.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2189768185' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77456af4-610d-4353-8774-41203640a91d"}]': finished 2024-04-06T03:12:42.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T03:12:42.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:12:42.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1221680699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:12:42.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:42 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:42.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2189768185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77456af4-610d-4353-8774-41203640a91d"}]: dispatch 2024-04-06T03:12:42.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2189768185' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77456af4-610d-4353-8774-41203640a91d"}]': finished 2024-04-06T03:12:42.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T03:12:42.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:12:42.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1221680699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:12:42.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:42 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:43.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:43 smithi060 ceph-mon[24896]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:43 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:12:43.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:43 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:43.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:43 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:43.829 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:43 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:43.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:43 smithi195 ceph-mon[27147]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:43 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:12:43.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:43 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:43.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:43 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:43.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:43 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:45.961 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:45.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:45.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:45.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:47.157 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:46 smithi060 ceph-mon[24896]: Reconfiguring mon.smithi060 (unknown last config time)... 2024-04-06T03:12:47.157 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:46 smithi060 ceph-mon[24896]: Reconfiguring daemon mon.smithi060 on smithi060 2024-04-06T03:12:47.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:46 smithi195 ceph-mon[27147]: Reconfiguring mon.smithi060 (unknown last config time)... 2024-04-06T03:12:47.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:46 smithi195 ceph-mon[27147]: Reconfiguring daemon mon.smithi060 on smithi060 2024-04-06T03:12:47.851 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:47 smithi060 ceph-mon[24896]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.852 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:47.852 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:12:47.852 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:12:47.852 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:48.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:47 smithi195 ceph-mon[27147]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:48.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:48.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:12:48.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:12:48.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:48.959 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:48 smithi060 ceph-mon[24896]: Reconfiguring mgr.smithi060.jttwce (unknown last config time)... 2024-04-06T03:12:48.959 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:48 smithi060 ceph-mon[24896]: Reconfiguring daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:12:49.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:48 smithi195 ceph-mon[27147]: Reconfiguring mgr.smithi060.jttwce (unknown last config time)... 2024-04-06T03:12:49.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:48 smithi195 ceph-mon[27147]: Reconfiguring daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:12:50.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:49 smithi060 ceph-mon[24896]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:50.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:49 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:12:50.107 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:49 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:50.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:49 smithi195 ceph-mon[27147]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:50.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:49 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:12:50.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:49 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:50.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:50 smithi060 ceph-mon[24896]: Deploying daemon osd.0 on smithi060 2024-04-06T03:12:50.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:51.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:50 smithi195 ceph-mon[27147]: Deploying daemon osd.0 on smithi060 2024-04-06T03:12:51.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:52.044 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:51 smithi060 ceph-mon[24896]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:52.044 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:51 smithi060 ceph-mon[24896]: Reconfiguring alertmanager.smithi060 (dependencies changed)... 2024-04-06T03:12:52.044 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:51 smithi060 ceph-mon[24896]: Reconfiguring daemon alertmanager.smithi060 on smithi060 2024-04-06T03:12:52.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:51 smithi195 ceph-mon[27147]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:52.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:51 smithi195 ceph-mon[27147]: Reconfiguring alertmanager.smithi060 (dependencies changed)... 2024-04-06T03:12:52.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:51 smithi195 ceph-mon[27147]: Reconfiguring daemon alertmanager.smithi060 on smithi060 2024-04-06T03:12:53.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:53 smithi060 ceph-mon[24896]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:54.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:53 smithi195 ceph-mon[27147]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:55.887 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:55 smithi060 ceph-mon[24896]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:55.887 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:56.078 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:55 smithi195 ceph-mon[27147]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.078 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:12:56.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:56 smithi060 ceph-mon[24896]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:57.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:56 smithi195 ceph-mon[27147]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:58.918 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:58 smithi060 ceph-mon[24896]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:58.918 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:12:58.918 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:12:58.918 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:58 smithi060 ceph-mon[24896]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:12:59.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:58 smithi195 ceph-mon[27147]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:59.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:12:59.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:12:59.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:58 smithi195 ceph-mon[27147]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:12:59.966 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:59 smithi060 ceph-mon[24896]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:12:59.966 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:59 smithi060 ceph-mon[24896]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T03:12:59.966 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:12:59.966 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:12:59 smithi060 ceph-mon[24896]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:13:00.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:59 smithi195 ceph-mon[27147]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:13:00.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:59 smithi195 ceph-mon[27147]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T03:13:00.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:00.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:12:59 smithi195 ceph-mon[27147]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:13:00.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:00 smithi060 ceph-mon[24896]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:00.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:00 smithi060 ceph-mon[24896]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:13:00.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:00 smithi060 ceph-mon[24896]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T03:13:00.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:00 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:00.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:00 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:00 smithi195 ceph-mon[27147]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:00 smithi195 ceph-mon[27147]: from='osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:13:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:00 smithi195 ceph-mon[27147]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T03:13:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:00 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:00 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438] boot 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T03:13:02.095 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:02.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:13:02.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:13:02.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:02.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: osd.0 [v2:172.21.15.60:6802/3771600438,v1:172.21.15.60:6803/3771600438] boot 2024-04-06T03:13:02.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T03:13:02.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: Reconfiguring crash.smithi060 (monmap changed)... 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:13:03.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:03.148 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:02 smithi060 ceph-mon[24896]: Reconfiguring daemon crash.smithi060 on smithi060 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: Reconfiguring crash.smithi060 (monmap changed)... 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:03.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:02 smithi195 ceph-mon[27147]: Reconfiguring daemon crash.smithi060 on smithi060 2024-04-06T03:13:03.809 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 0 on host 'smithi060' 2024-04-06T03:13:05.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:04 smithi060 ceph-mon[24896]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:05.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:05.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:04 smithi195 ceph-mon[27147]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:05.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:05.469 DEBUG:teuthology.orchestra.run.smithi060:osd.0> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-06T03:13:05.471 INFO:tasks.cephadm:Deploying osd.1 on smithi060 with /dev/vg_nvme/lv_3... 2024-04-06T03:13:05.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T03:13:06.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:06 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:06.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:06 smithi195 ceph-mon[27147]: Reconfiguring grafana.smithi060 (dependencies changed)... 2024-04-06T03:13:06.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:06 smithi195 ceph-mon[27147]: Reconfiguring daemon grafana.smithi060 on smithi060 2024-04-06T03:13:06.478 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:06 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:06.478 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:06 smithi060 ceph-mon[24896]: Reconfiguring grafana.smithi060 (dependencies changed)... 2024-04-06T03:13:06.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:06 smithi060 ceph-mon[24896]: Reconfiguring daemon grafana.smithi060 on smithi060 2024-04-06T03:13:07.461 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:07 smithi195 ceph-mon[27147]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:07.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:07 smithi060 ceph-mon[24896]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:09.375 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:09 smithi060 ceph-mon[24896]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:09.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:09 smithi195 ceph-mon[27147]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00879725 s, 1.2 GB/s 2024-04-06T03:13:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: Reconfiguring crash.smithi195 (monmap changed)... 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:10 smithi060 ceph-mon[24896]: Reconfiguring daemon crash.smithi195 on smithi195 2024-04-06T03:13:11.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi060:vg_nvme/lv_3 2024-04-06T03:13:11.172 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:11.173 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:11.173 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: Reconfiguring crash.smithi195 (monmap changed)... 2024-04-06T03:13:11.173 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:13:11.173 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:11.173 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:10 smithi195 ceph-mon[27147]: Reconfiguring daemon crash.smithi195 on smithi195 2024-04-06T03:13:12.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:12.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: Reconfiguring mgr.smithi195.ultkot (monmap changed)... 2024-04-06T03:13:12.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:12.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:12.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:12.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:12 smithi060 ceph-mon[24896]: Reconfiguring daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:13:12.486 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:12.486 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: Reconfiguring mgr.smithi195.ultkot (monmap changed)... 2024-04-06T03:13:12.486 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:12.486 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:12.486 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:12.487 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:12 smithi195 ceph-mon[27147]: Reconfiguring daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:13:13.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:13.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:13.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: Reconfiguring mon.smithi195 (monmap changed)... 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: Reconfiguring daemon mon.smithi195 on smithi195 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:13.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: Reconfiguring mon.smithi195 (monmap changed)... 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: Reconfiguring daemon mon.smithi195 on smithi195 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:13.712 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:13.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:14.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:14 smithi060 ceph-mon[24896]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:14.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:14 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:14.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:14 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:13:14.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:14 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:13:14.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:14 smithi195 ceph-mon[27147]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:14.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:14 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:14.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:14 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:13:14.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:14 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:13:15.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:15 smithi195 ceph-mon[27147]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:15.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:15 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3352371118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecb76e-20b2-4e31-9d59-c3c5120183f8"}]: dispatch 2024-04-06T03:13:15.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:15 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3352371118' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99ecb76e-20b2-4e31-9d59-c3c5120183f8"}]': finished 2024-04-06T03:13:15.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:15 smithi195 ceph-mon[27147]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T03:13:15.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:15 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:15.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:15 smithi060 ceph-mon[24896]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:15.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:15 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3352371118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecb76e-20b2-4e31-9d59-c3c5120183f8"}]: dispatch 2024-04-06T03:13:15.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:15 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3352371118' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99ecb76e-20b2-4e31-9d59-c3c5120183f8"}]': finished 2024-04-06T03:13:15.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:15 smithi060 ceph-mon[24896]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T03:13:15.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:15 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:17.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:16 smithi195 ceph-mon[27147]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:17.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:16 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/84274689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:17.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:16 smithi060 ceph-mon[24896]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:17.238 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:16 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/84274689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:18.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:18 smithi060 ceph-mon[24896]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:19.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:18 smithi195 ceph-mon[27147]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:20.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:20 smithi060 ceph-mon[24896]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:21.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:20 smithi195 ceph-mon[27147]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:21.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:21 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:13:21.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:21 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:21.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:21 smithi060 ceph-mon[24896]: Deploying daemon osd.1 on smithi060 2024-04-06T03:13:22.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:21 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:13:22.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:21 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:22.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:21 smithi195 ceph-mon[27147]: Deploying daemon osd.1 on smithi060 2024-04-06T03:13:23.163 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:22 smithi060 ceph-mon[24896]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:23.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:22 smithi195 ceph-mon[27147]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:24.949 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:24 smithi060 ceph-mon[24896]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:25.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:24 smithi195 ceph-mon[27147]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:27.813 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:27 smithi060 ceph-mon[24896]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:27.813 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:27.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:27 smithi195 ceph-mon[27147]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:27.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:27 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.194 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 1 on host 'smithi060' 2024-04-06T03:13:29.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:29.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.60:9093"}]: dispatch 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.60:3000"}]: dispatch 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.60:9095"}]: dispatch 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.792 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:29.793 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:29.793 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:29.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:29.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.60:9093"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.60:3000"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.60:9095"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:29.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:29.914 DEBUG:teuthology.orchestra.run.smithi060:osd.1> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-06T03:13:29.916 INFO:tasks.cephadm:Deploying osd.2 on smithi060 with /dev/vg_nvme/lv_2... 2024-04-06T03:13:29.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.60:9093"}]: dispatch 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.60:3000"}]: dispatch 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.60:9095"}]: dispatch 2024-04-06T03:13:30.826 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:30 smithi195 ceph-mon[27147]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:13:30.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.60:9093"}]: dispatch 2024-04-06T03:13:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:13:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.60:3000"}]: dispatch 2024-04-06T03:13:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:13:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.60:9095"}]: dispatch 2024-04-06T03:13:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:30 smithi060 ceph-mon[24896]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:13:31.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:31.652 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[38196]: 2024-04-06T03:13:31.550+0000 7f19e8114700 -1 osd.1 0 waiting for initial osdmap 2024-04-06T03:13:31.652 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:13:31 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[38196]: 2024-04-06T03:13:31.553+0000 7f19e0aa7700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:13:31.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:32.842 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:32 smithi060 ceph-mon[24896]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:13:32.842 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:32 smithi060 ceph-mon[24896]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T03:13:32.842 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:32 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:32.842 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:32 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:32.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:32 smithi195 ceph-mon[27147]: from='osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:13:32.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:32 smithi195 ceph-mon[27147]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T03:13:32.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:32 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:32.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:32 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T03:13:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T03:13:33.160 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-04-06T03:13:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-04-06T03:13:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00953767 s, 1.1 GB/s 2024-04-06T03:13:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135] boot 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T03:13:33.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:33.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi060:vg_nvme/lv_2 2024-04-06T03:13:33.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:13:33.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:13:33.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:13:33.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: osd.1 [v2:172.21.15.60:6810/3010589135,v1:172.21.15.60:6811/3010589135] boot 2024-04-06T03:13:33.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T03:13:33.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:13:34.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:34 smithi195 ceph-mon[27147]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T03:13:34.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:34 smithi060 ceph-mon[24896]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T03:13:35.645 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:35 smithi060 ceph-mon[24896]: pgmap v51: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:35.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:35 smithi195 ceph-mon[27147]: pgmap v51: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:37.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:36 smithi060 ceph-mon[24896]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:37.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:36 smithi060 ceph-mon[24896]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:37.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:36 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:37.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:36 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:37.108 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:36 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:37.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:36 smithi195 ceph-mon[27147]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:37.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:36 smithi195 ceph-mon[27147]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:37.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:36 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:37.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:36 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:37.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:36 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:38.173 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:38.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:39.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:39 smithi195 ceph-mon[27147]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:39.489 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:39 smithi060 ceph-mon[24896]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:40.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:40 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/278494870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d"}]: dispatch 2024-04-06T03:13:40.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:40 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/278494870' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d"}]': finished 2024-04-06T03:13:40.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:40 smithi195 ceph-mon[27147]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T03:13:40.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:13:40.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:40 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/278494870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d"}]: dispatch 2024-04-06T03:13:40.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:40 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/278494870' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d"}]': finished 2024-04-06T03:13:40.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:40 smithi060 ceph-mon[24896]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T03:13:40.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:13:41.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:41 smithi195 ceph-mon[27147]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:41.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:41 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1163692461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:41.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:41 smithi060 ceph-mon[24896]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:41.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:41 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1163692461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:43.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:43 smithi060 ceph-mon[24896]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:43.462 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:43 smithi195 ceph-mon[27147]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:45.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:45 smithi060 ceph-mon[24896]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:45.414 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:45 smithi195 ceph-mon[27147]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi060 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi060 to 6434M 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:46.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:46.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:46 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:47.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi060 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi060 to 6434M 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:47.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:46 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:48.206 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:13:48.206 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:48.206 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:47 smithi060 ceph-mon[24896]: Deploying daemon osd.2 on smithi060 2024-04-06T03:13:48.207 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:48.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:13:48.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:48.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:47 smithi195 ceph-mon[27147]: Deploying daemon osd.2 on smithi060 2024-04-06T03:13:48.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:49.809 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:49 smithi060 ceph-mon[24896]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:49.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:49 smithi195 ceph-mon[27147]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:51.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:51 smithi060 ceph-mon[24896]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:51.962 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:51 smithi195 ceph-mon[27147]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:53.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:53 smithi060 ceph-mon[24896]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:53.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:53 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:54.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:53 smithi195 ceph-mon[27147]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:54.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:53 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:55.880 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:55 smithi060 ceph-mon[24896]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:55.880 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:55.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:55 smithi195 ceph-mon[27147]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:55.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:57.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:57.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:58.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:13:58.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:13:58.994 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:13:58.994 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:13:59.510 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 2 on host 'smithi060' 2024-04-06T03:13:59.865 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:59 smithi060 ceph-mon[24896]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:13:59.865 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:59 smithi060 ceph-mon[24896]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:13:59.865 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:13:59.865 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:13:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:00.163 DEBUG:teuthology.orchestra.run.smithi060:osd.2> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-06T03:14:00.166 INFO:tasks.cephadm:Deploying osd.3 on smithi060 with /dev/vg_nvme/lv_1... 2024-04-06T03:14:00.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:59 smithi195 ceph-mon[27147]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:00.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:59 smithi195 ceph-mon[27147]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:14:00.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:00.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:13:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:00.947 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-mon[24896]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:14:00.948 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-mon[24896]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T03:14:00.948 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:00.948 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-mon[24896]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:14:00.948 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[42063]: 2024-04-06T03:14:00.738+0000 7f06badb8700 -1 osd.2 0 waiting for initial osdmap 2024-04-06T03:14:00.948 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:00 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[42063]: 2024-04-06T03:14:00.744+0000 7f06b5f50700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:14:01.212 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:00 smithi195 ceph-mon[27147]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:14:01.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:00 smithi195 ceph-mon[27147]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T03:14:01.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:00 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:01.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:00 smithi195 ceph-mon[27147]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:14:01.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:01 smithi060 ceph-mon[24896]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:01.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:01 smithi060 ceph-mon[24896]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:14:01.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:01 smithi060 ceph-mon[24896]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T03:14:01.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:01.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:02.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:01 smithi195 ceph-mon[27147]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:02.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:01 smithi195 ceph-mon[27147]: from='osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:14:02.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:01 smithi195 ceph-mon[27147]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T03:14:02.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:02.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:02.979 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:14:02.979 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:14:02.979 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:02.979 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721] boot 2024-04-06T03:14:02.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T03:14:02.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:02.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: osd.2 [v2:172.21.15.60:6818/1209852721,v1:172.21.15.60:6819/1209852721] boot 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:14:03.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T03:14:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T03:14:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T03:14:03.576 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-04-06T03:14:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-04-06T03:14:03.576 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00953052 s, 1.1 GB/s 2024-04-06T03:14:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-04-06T03:14:03.907 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:03 smithi060 ceph-mon[24896]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:03.907 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T03:14:03.907 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:03 smithi060 ceph-mon[24896]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T03:14:03.907 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:14:04.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:03 smithi195 ceph-mon[27147]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:04.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T03:14:04.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:03 smithi195 ceph-mon[27147]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T03:14:04.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:14:04.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi060:vg_nvme/lv_1 2024-04-06T03:14:04.953 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:14:04.953 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:04 smithi060 ceph-mon[24896]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T03:14:04.954 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43449]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T03:14:04.954 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43449]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:04.954 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43449]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:04.954 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43463]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T03:14:05.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:14:05.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:04 smithi195 ceph-mon[27147]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T03:14:05.237 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43463]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:05.237 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:14:04 smithi060 sudo[43463]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:05.489 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43504]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T03:14:05.489 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43504]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:05.489 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43504]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:05.761 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43527]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T03:14:05.762 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43527]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:05.762 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43527]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:05.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:05 smithi060 ceph-mon[24896]: pgmap v72: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:05.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:05 smithi060 ceph-mon[24896]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T03:14:05.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:05 smithi060 ceph-mon[24896]: mgrmap e19: smithi060.jttwce(active, since 2m), standbys: smithi195.ultkot 2024-04-06T03:14:05.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:05.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:05 smithi195 ceph-mon[27147]: pgmap v72: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:05.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:05 smithi195 ceph-mon[27147]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T03:14:05.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:05 smithi195 ceph-mon[27147]: mgrmap e19: smithi060.jttwce(active, since 2m), standbys: smithi195.ultkot 2024-04-06T03:14:05.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:05 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:06.027 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43531]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T03:14:06.027 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43531]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:06.027 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:05 smithi060 sudo[43531]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:06.288 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43534]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T03:14:06.288 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43534]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:06.288 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43534]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:06.542 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43570]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T03:14:06.542 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43570]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:06.542 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43570]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:06.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43583]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T03:14:06.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43583]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:06.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:06 smithi060 sudo[43583]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:06 smithi195 sudo[29470]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:06 smithi195 sudo[29470]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:06 smithi195 sudo[29470]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 sudo[29473]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 sudo[29473]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T03:14:07.060 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 sudo[29473]: pam_unix(sudo:session): session closed for user root 2024-04-06T03:14:07.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:07.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:14:07.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:14:07.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:14:07.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:14:07.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:07.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:14:07.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:07 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:14:07.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:07 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:08.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:14:08.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:08 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:14:08.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:08 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:09.396 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:09 smithi060 ceph-mon[24896]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:09.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:09 smithi195 ceph-mon[27147]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi060 to 4289M 2024-04-06T03:14:11.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:11.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:11.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:11 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:11.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:11.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi060 to 4289M 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:11 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:12.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:12 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/787501546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b68683e8-78ed-4864-bda4-461df17fa0fc"}]: dispatch 2024-04-06T03:14:12.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:12 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/787501546' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b68683e8-78ed-4864-bda4-461df17fa0fc"}]': finished 2024-04-06T03:14:12.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:12 smithi195 ceph-mon[27147]: osdmap e23: 4 total, 3 up, 4 in 2024-04-06T03:14:12.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:12.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:12 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/4028125192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:12.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:12 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/787501546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b68683e8-78ed-4864-bda4-461df17fa0fc"}]: dispatch 2024-04-06T03:14:12.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:12 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/787501546' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b68683e8-78ed-4864-bda4-461df17fa0fc"}]': finished 2024-04-06T03:14:12.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:12 smithi060 ceph-mon[24896]: osdmap e23: 4 total, 3 up, 4 in 2024-04-06T03:14:12.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:12.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:12 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/4028125192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:13.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:13 smithi060 ceph-mon[24896]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:13.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:13 smithi195 ceph-mon[27147]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:15.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:15 smithi195 ceph-mon[27147]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:15.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:15 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:15.718 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:15 smithi060 ceph-mon[24896]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:15.718 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:15 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:16.691 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:16 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:16.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:16 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:17.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:17 smithi195 ceph-mon[27147]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:17.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:17 smithi060 ceph-mon[24896]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:18.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:18 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:14:18.713 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:18 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:18.714 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:18 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:14:18.714 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:18 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:19.537 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:19 smithi060 ceph-mon[24896]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:19.538 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:19 smithi060 ceph-mon[24896]: Deploying daemon osd.3 on smithi060 2024-04-06T03:14:19.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:19 smithi195 ceph-mon[27147]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:19.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:19 smithi195 ceph-mon[27147]: Deploying daemon osd.3 on smithi060 2024-04-06T03:14:21.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:21 smithi060 ceph-mon[24896]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:21.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:21 smithi195 ceph-mon[27147]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:23.653 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:23 smithi060 ceph-mon[24896]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:23.653 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:23.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:23.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:23.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:23 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:23.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:23 smithi195 ceph-mon[27147]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:23.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:23 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:23.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:23 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:23.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:23 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:23.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:23 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:25.650 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:25 smithi195 ceph-mon[27147]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:25.726 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:25 smithi060 ceph-mon[24896]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:26.731 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 3 on host 'smithi060' 2024-04-06T03:14:26.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:26 smithi060 ceph-mon[24896]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:26.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:26 smithi060 ceph-mon[24896]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:14:26.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:26 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:27.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:26 smithi195 ceph-mon[27147]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:27.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:26 smithi195 ceph-mon[27147]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:14:27.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:26 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:27.335 DEBUG:teuthology.orchestra.run.smithi060:osd.3> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-06T03:14:27.338 INFO:tasks.cephadm:Deploying osd.4 on smithi195 with /dev/vg_nvme/lv_4... 2024-04-06T03:14:27.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T03:14:27.947 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:27 smithi195 ceph-mon[27147]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:14:27.947 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:27 smithi195 ceph-mon[27147]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T03:14:27.948 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:27 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:27.948 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:27 smithi195 ceph-mon[27147]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:14:28.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-mon[24896]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:14:28.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-mon[24896]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T03:14:28.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:28.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-mon[24896]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:14:28.104 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[45896]: 2024-04-06T03:14:27.841+0000 7ffac1f8f700 -1 osd.3 0 waiting for initial osdmap 2024-04-06T03:14:28.104 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:14:27 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[45896]: 2024-04-06T03:14:27.845+0000 7ffabd127700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:14:28.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:28 smithi195 ceph-mon[27147]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:28.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:28 smithi195 ceph-mon[27147]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:14:28.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:28 smithi195 ceph-mon[27147]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T03:14:28.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:28.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:28 smithi060 ceph-mon[24896]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:28 smithi060 ceph-mon[24896]: from='osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-06T03:14:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:28 smithi060 ceph-mon[24896]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T03:14:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:29.182 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850645 s, 1.2 GB/s 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 2024-04-06T03:14:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-06T03:14:29.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi195:vg_nvme/lv_4 2024-04-06T03:14:29.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:29 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:14:29.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:29 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:14:29.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:29 smithi195 ceph-mon[27147]: osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869] boot 2024-04-06T03:14:29.963 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:29 smithi195 ceph-mon[27147]: osdmap e26: 4 total, 4 up, 4 in 2024-04-06T03:14:29.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:29 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:30.239 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:29 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:14:30.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:29 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:14:30.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:29 smithi060 ceph-mon[24896]: osd.3 [v2:172.21.15.60:6826/2629454869,v1:172.21.15.60:6827/2629454869] boot 2024-04-06T03:14:30.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:29 smithi060 ceph-mon[24896]: osdmap e26: 4 total, 4 up, 4 in 2024-04-06T03:14:30.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:29 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:14:30.989 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:30 smithi060 ceph-mon[24896]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:30.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:30 smithi060 ceph-mon[24896]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T03:14:30.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:30 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:31.192 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:30 smithi195 ceph-mon[27147]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:14:31.192 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:30 smithi195 ceph-mon[27147]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T03:14:31.192 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:30 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:31.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:31 smithi060 ceph-mon[24896]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:31.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:31.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:31.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:32.116 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:31 smithi195 ceph-mon[27147]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:32.116 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:32.116 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:32.116 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:32.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:32 smithi060 ceph-mon[24896]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:33.123 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:32 smithi195 ceph-mon[27147]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi060 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi060 to 3217M 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3447106259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]: dispatch 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]: dispatch 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]': finished 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: osdmap e28: 5 total, 4 up, 5 in 2024-04-06T03:14:34.407 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:34 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:34.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi060 2024-04-06T03:14:34.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi060 to 3217M 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/3447106259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]: dispatch 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa47800-56f1-42f2-87cd-d16421d3ed08"}]': finished 2024-04-06T03:14:34.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: osdmap e28: 5 total, 4 up, 5 in 2024-04-06T03:14:34.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:34 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:35.353 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:35 smithi060 ceph-mon[24896]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:35.353 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:35 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3174856563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:35.463 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:35 smithi195 ceph-mon[27147]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:35.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:35 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/3174856563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:37.354 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:37 smithi195 ceph-mon[27147]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:37.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:37 smithi060 ceph-mon[24896]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:38.394 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:38.394 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:38.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:38.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:39.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:39 smithi195 ceph-mon[27147]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:39.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:39 smithi060 ceph-mon[24896]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:40.445 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:14:40.445 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:40.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:14:40.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:41.461 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:41 smithi195 ceph-mon[27147]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:41.461 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:41 smithi195 ceph-mon[27147]: Deploying daemon osd.4 on smithi195 2024-04-06T03:14:41.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:41 smithi060 ceph-mon[24896]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:41.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:41 smithi060 ceph-mon[24896]: Deploying daemon osd.4 on smithi195 2024-04-06T03:14:43.400 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:43 smithi195 ceph-mon[27147]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:43.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:43 smithi060 ceph-mon[24896]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:45.213 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:45 smithi195 ceph-mon[27147]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:45.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:45.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:45.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:45.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:45.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:45 smithi060 ceph-mon[24896]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:45.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:45.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:45.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:45.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:47.214 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:47 smithi195 ceph-mon[27147]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:47.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:47 smithi060 ceph-mon[24896]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:47.787 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 4 on host 'smithi195' 2024-04-06T03:14:48.374 DEBUG:teuthology.orchestra.run.smithi195:osd.4> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-06T03:14:48.377 INFO:tasks.cephadm:Deploying osd.5 on smithi195 with /dev/vg_nvme/lv_3... 2024-04-06T03:14:48.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T03:14:49.098 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:48 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:49.098 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:48 smithi195 ceph-mon[27147]: from='osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:14:49.098 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:48 smithi195 ceph-mon[27147]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:14:49.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:48 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:49.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:48 smithi060 ceph-mon[24896]: from='osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:14:49.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:48 smithi060 ceph-mon[24896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:14:50.164 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:50.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:14:50.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: osdmap e29: 5 total, 4 up, 5 in 2024-04-06T03:14:50.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:50.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: from='osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:14:50.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-mon[27147]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:14:50.165 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[31403]: 2024-04-06T03:14:49.805+0000 7f05c6454700 -1 osd.4 0 waiting for initial osdmap 2024-04-06T03:14:50.165 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:14:49 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[31403]: 2024-04-06T03:14:49.809+0000 7f05c0deb700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: osdmap e29: 5 total, 4 up, 5 in 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: from='osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:14:50.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:49 smithi060 ceph-mon[24896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0106567 s, 984 MB/s 2024-04-06T03:14:50.628 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-06T03:14:51.101 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:50 smithi195 ceph-mon[27147]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:14:51.101 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:50 smithi195 ceph-mon[27147]: osdmap e30: 5 total, 4 up, 5 in 2024-04-06T03:14:51.101 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.101 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:50 smithi060 ceph-mon[24896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:14:51.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:50 smithi060 ceph-mon[24896]: osdmap e30: 5 total, 4 up, 5 in 2024-04-06T03:14:51.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi195:vg_nvme/lv_3 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464] boot 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: osdmap e31: 5 total, 5 up, 5 in 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:51.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: osdmap e32: 5 total, 5 up, 5 in 2024-04-06T03:14:51.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:52.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:14:52.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:14:52.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T03:14:52.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:52.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: osd.4 [v2:172.21.15.195:6800/433073464,v1:172.21.15.195:6801/433073464] boot 2024-04-06T03:14:52.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: osdmap e31: 5 total, 5 up, 5 in 2024-04-06T03:14:52.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:14:52.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: osdmap e32: 5 total, 5 up, 5 in 2024-04-06T03:14:52.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:53.096 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:52 smithi195 ceph-mon[27147]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:14:53.096 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:52 smithi195 ceph-mon[27147]: osdmap e33: 5 total, 5 up, 5 in 2024-04-06T03:14:53.141 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:52 smithi060 ceph-mon[24896]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:14:53.141 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:52 smithi060 ceph-mon[24896]: osdmap e33: 5 total, 5 up, 5 in 2024-04-06T03:14:54.101 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:53 smithi195 ceph-mon[27147]: osdmap e34: 5 total, 5 up, 5 in 2024-04-06T03:14:54.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:53 smithi060 ceph-mon[24896]: osdmap e34: 5 total, 5 up, 5 in 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:54.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:55.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-06T03:14:55.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T03:14:55.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:55.240 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:55.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:55.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi195 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi195 to 15936M 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/639266032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]: dispatch 2024-04-06T03:14:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:57 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]: dispatch 2024-04-06T03:14:57.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T03:14:57.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi195 2024-04-06T03:14:57.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi195 to 15936M 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/639266032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]: dispatch 2024-04-06T03:14:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:57 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]: dispatch 2024-04-06T03:14:58.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]': finished 2024-04-06T03:14:58.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: osdmap e36: 6 total, 5 up, 6 in 2024-04-06T03:14:58.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:14:58.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/3287093562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:58.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:14:58.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4ba76c8-b1e0-43ab-be8d-ce5d564573e3"}]': finished 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: osdmap e36: 6 total, 5 up, 6 in 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3287093562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:14:58.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:14:59.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:14:59 smithi195 ceph-mon[27147]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T03:14:59.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:14:59 smithi060 ceph-mon[24896]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T03:15:01.334 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:01 smithi195 ceph-mon[27147]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-06T03:15:01.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:01 smithi060 ceph-mon[24896]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-06T03:15:03.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:03 smithi060 ceph-mon[24896]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:03.596 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:03 smithi195 ceph-mon[27147]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:04.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:15:04.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:04.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:04 smithi060 ceph-mon[24896]: Deploying daemon osd.5 on smithi195 2024-04-06T03:15:04.525 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:15:04.525 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:04.525 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:04 smithi195 ceph-mon[27147]: Deploying daemon osd.5 on smithi195 2024-04-06T03:15:05.423 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:05 smithi195 ceph-mon[27147]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:05.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:05 smithi060 ceph-mon[24896]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:07.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:07 smithi060 ceph-mon[24896]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:07.589 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:07 smithi195 ceph-mon[27147]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:09.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:09 smithi060 ceph-mon[24896]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:09.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:09 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:09.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:09 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:09.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:09 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:09.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:09 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:09.807 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:09 smithi195 ceph-mon[27147]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:09.807 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:09 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:09.807 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:09 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:09.807 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:09 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:09.808 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:09 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:11.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:11 smithi060 ceph-mon[24896]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:11.945 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:11 smithi195 ceph-mon[27147]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:12.242 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 5 on host 'smithi195' 2024-04-06T03:15:12.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:12 smithi060 ceph-mon[24896]: from='osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:15:12.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:12 smithi060 ceph-mon[24896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:15:12.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:12 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:12.815 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:12 smithi195 ceph-mon[27147]: from='osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:15:12.815 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:12 smithi195 ceph-mon[27147]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:15:12.815 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:12 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:12.838 DEBUG:teuthology.orchestra.run.smithi195:osd.5> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-06T03:15:12.840 INFO:tasks.cephadm:Deploying osd.6 on smithi195 with /dev/vg_nvme/lv_2... 2024-04-06T03:15:12.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T03:15:13.829 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:13.829 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:15:13.829 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T03:15:13.830 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:13.830 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: from='osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:13.830 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:13.830 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:13.830 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[34374]: 2024-04-06T03:15:13.503+0000 7f67ee6b4700 -1 osd.5 0 waiting for initial osdmap 2024-04-06T03:15:13.830 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:15:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[34374]: 2024-04-06T03:15:13.507+0000 7f67e984c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:15:13.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:13.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:15:13.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T03:15:13.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:13.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: from='osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:13.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:13.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:13 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:14.750 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:14 smithi195 ceph-mon[27147]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:15:14.751 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:14 smithi195 ceph-mon[27147]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T03:15:14.751 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:14 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:14.751 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:14 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:14.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:14 smithi060 ceph-mon[24896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:15:14.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:14 smithi060 ceph-mon[24896]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T03:15:14.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:14 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:14.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:14 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013] boot 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: osdmap e39: 6 total, 6 up, 6 in 2024-04-06T03:15:15.651 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:15 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:15.827 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T03:15:15.827 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T03:15:15.827 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-06T03:15:15.828 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-06T03:15:15.828 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00981046 s, 1.1 GB/s 2024-04-06T03:15:15.828 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-06T03:15:15.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: osd.5 [v2:172.21.15.195:6808/1622341013,v1:172.21.15.195:6809/1622341013] boot 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: osdmap e39: 6 total, 6 up, 6 in 2024-04-06T03:15:15.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:15 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:15:16.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi195:vg_nvme/lv_2 2024-04-06T03:15:16.942 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:16 smithi195 ceph-mon[27147]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T03:15:16.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:16 smithi060 ceph-mon[24896]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T03:15:17.538 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:17 smithi195 ceph-mon[27147]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:17.538 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:17 smithi195 ceph-mon[27147]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T03:15:17.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:17 smithi060 ceph-mon[24896]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:17.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:17 smithi060 ceph-mon[24896]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T03:15:18.818 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:18 smithi195 ceph-mon[27147]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T03:15:18.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:18 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:18.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:18 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:18.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:18 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:18.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:18 smithi060 ceph-mon[24896]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T03:15:18.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:18 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:18.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:18 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:18.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:18 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:19.854 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi195 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi195 to 7968M 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.855 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.856 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:19.856 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:19.856 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:19.856 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi195 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi195 to 7968M 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:19.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/3123614740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]: dispatch 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]: dispatch 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]': finished 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: osdmap e44: 7 total, 6 up, 7 in 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:21.965 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:21 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/1317556434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:21.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:21.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/3123614740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]: dispatch 2024-04-06T03:15:21.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]: dispatch 2024-04-06T03:15:21.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbd4aaac-1089-4046-8880-4aa726906692"}]': finished 2024-04-06T03:15:21.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: osdmap e44: 7 total, 6 up, 7 in 2024-04-06T03:15:21.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:21.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:21 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/1317556434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:23.793 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:23 smithi060 ceph-mon[24896]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:23.964 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:23 smithi195 ceph-mon[27147]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:25.707 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:25 smithi195 ceph-mon[27147]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:25.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:25 smithi060 ceph-mon[24896]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:27.867 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:27 smithi195 ceph-mon[27147]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:27.868 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:27 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:15:27.868 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:27 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:27.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:27 smithi060 ceph-mon[24896]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:27.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:15:27.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:28.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:28 smithi195 ceph-mon[27147]: Deploying daemon osd.6 on smithi195 2024-04-06T03:15:28.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:28 smithi060 ceph-mon[24896]: Deploying daemon osd.6 on smithi195 2024-04-06T03:15:29.870 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:29 smithi195 ceph-mon[27147]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:29.991 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:29 smithi060 ceph-mon[24896]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:31.865 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:31 smithi195 ceph-mon[27147]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:31.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:31 smithi060 ceph-mon[24896]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:33.435 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:33 smithi195 ceph-mon[27147]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:33.436 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:33.436 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:33.436 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:33.436 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:33.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:33 smithi060 ceph-mon[24896]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:33.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:33.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:33.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:33.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:35.039 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 6 on host 'smithi195' 2024-04-06T03:15:35.442 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:35 smithi195 ceph-mon[27147]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:35.442 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:35.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:35 smithi060 ceph-mon[24896]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:35.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:35.672 DEBUG:teuthology.orchestra.run.smithi195:osd.6> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-06T03:15:35.675 INFO:tasks.cephadm:Deploying osd.7 on smithi195 with /dev/vg_nvme/lv_1... 2024-04-06T03:15:35.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T03:15:36.346 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:36 smithi195 ceph-mon[27147]: from='osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:15:36.346 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:36 smithi195 ceph-mon[27147]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:15:36.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:36 smithi060 ceph-mon[24896]: from='osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:15:36.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:36 smithi060 ceph-mon[24896]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: osdmap e45: 7 total, 6 up, 7 in 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: from='osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:37.326 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-mon[27147]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:37.327 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[37359]: 2024-04-06T03:15:37.130+0000 7f5dcb088700 -1 osd.6 0 waiting for initial osdmap 2024-04-06T03:15:37.327 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:15:37 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[37359]: 2024-04-06T03:15:37.137+0000 7f5dc5a1f700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: osdmap e45: 7 total, 6 up, 7 in 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: from='osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:37.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:37 smithi060 ceph-mon[24896]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00934112 s, 1.1 GB/s 2024-04-06T03:15:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-06T03:15:38.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:38 smithi195 ceph-mon[27147]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:15:38.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:38 smithi195 ceph-mon[27147]: osdmap e46: 7 total, 6 up, 7 in 2024-04-06T03:15:38.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:38.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:38.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:38.490 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:38 smithi060 ceph-mon[24896]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:15:38.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:38 smithi060 ceph-mon[24896]: osdmap e46: 7 total, 6 up, 7 in 2024-04-06T03:15:38.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:38.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:38.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:38.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi195:vg_nvme/lv_1 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813] boot 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: osdmap e47: 7 total, 7 up, 7 in 2024-04-06T03:15:39.328 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:39 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: osd.6 [v2:172.21.15.195:6816/2952424813,v1:172.21.15.195:6817/2952424813] boot 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: osdmap e47: 7 total, 7 up, 7 in 2024-04-06T03:15:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:39 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:15:40.464 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:40 smithi195 ceph-mon[27147]: osdmap e48: 7 total, 7 up, 7 in 2024-04-06T03:15:40.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:40 smithi060 ceph-mon[24896]: osdmap e48: 7 total, 7 up, 7 in 2024-04-06T03:15:41.401 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:41 smithi195 ceph-mon[27147]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:41.401 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:41 smithi195 ceph-mon[27147]: osdmap e49: 7 total, 7 up, 7 in 2024-04-06T03:15:41.401 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:41 smithi195 ceph-mon[27147]: osdmap e50: 7 total, 7 up, 7 in 2024-04-06T03:15:41.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:41 smithi060 ceph-mon[24896]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:41.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:41 smithi060 ceph-mon[24896]: osdmap e49: 7 total, 7 up, 7 in 2024-04-06T03:15:41.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:41 smithi060 ceph-mon[24896]: osdmap e50: 7 total, 7 up, 7 in 2024-04-06T03:15:42.357 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:42 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:42.358 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:42 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:42.358 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:42 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:42.358 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:42 smithi195 ceph-mon[27147]: osdmap e51: 7 total, 7 up, 7 in 2024-04-06T03:15:42.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:42 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:42.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:42 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:42.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:42 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:42.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:42 smithi060 ceph-mon[24896]: osdmap e51: 7 total, 7 up, 7 in 2024-04-06T03:15:43.338 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:43 smithi195 ceph-mon[27147]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:43.338 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:43 smithi195 ceph-mon[27147]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:43.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:43 smithi060 ceph-mon[24896]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:43.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:43 smithi060 ceph-mon[24896]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:44.715 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi195 2024-04-06T03:15:44.715 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.715 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi195 to 5312M 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:44.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:44 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi195 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi195 to 5312M 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:44.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:44.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:44.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:44 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/1011108436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]: dispatch 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]: dispatch 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]': finished 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: osdmap e52: 8 total, 7 up, 8 in 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:15:45.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:45 smithi195 ceph-mon[27147]: from='client.? 172.21.15.195:0/67730194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:45.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-06T03:15:45.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/1011108436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]: dispatch 2024-04-06T03:15:45.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]: dispatch 2024-04-06T03:15:45.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd6732b5-ce77-48a8-80f1-688a0aa1044f"}]': finished 2024-04-06T03:15:45.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: osdmap e52: 8 total, 7 up, 8 in 2024-04-06T03:15:45.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:15:45.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:45 smithi060 ceph-mon[24896]: from='client.? 172.21.15.195:0/67730194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:47.714 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:47 smithi195 ceph-mon[27147]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-06T03:15:47.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:47 smithi060 ceph-mon[24896]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-06T03:15:49.614 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:49 smithi195 ceph-mon[27147]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-06T03:15:49.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:49 smithi060 ceph-mon[24896]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-06T03:15:51.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:51 smithi195 ceph-mon[27147]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T03:15:51.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:15:51.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:51.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:51 smithi060 ceph-mon[24896]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T03:15:51.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:15:51.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:52.596 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:52 smithi195 ceph-mon[27147]: Deploying daemon osd.7 on smithi195 2024-04-06T03:15:52.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:52 smithi060 ceph-mon[24896]: Deploying daemon osd.7 on smithi195 2024-04-06T03:15:53.715 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:53 smithi195 ceph-mon[27147]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T03:15:53.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:53 smithi060 ceph-mon[24896]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T03:15:55.715 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:55 smithi195 ceph-mon[27147]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:55.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:55 smithi060 ceph-mon[24896]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:57 smithi195 ceph-mon[27147]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:57.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:57 smithi060 ceph-mon[24896]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:15:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:57.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:58.393 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:15:58.393 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:15:58.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:15:58.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:15:59.324 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:15:59 smithi195 ceph-mon[27147]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:59.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:15:59 smithi060 ceph-mon[24896]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:15:59.631 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 7 on host 'smithi195' 2024-04-06T03:16:00.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:00 smithi195 ceph-mon[27147]: from='osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:16:00.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:00 smithi195 ceph-mon[27147]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:16:00.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:00 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:00.309 DEBUG:teuthology.orchestra.run.smithi195:osd.7> sudo journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-06T03:16:00.312 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-06T03:16:00.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-06T03:16:00.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:00 smithi060 ceph-mon[24896]: from='osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:16:00.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:00 smithi060 ceph-mon[24896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:16:00.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:00 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:01.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:16:01.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:16:01.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: osdmap e53: 8 total, 7 up, 8 in 2024-04-06T03:16:01.242 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:01.242 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: from='osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:16:01.242 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:16:01.242 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:01 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:01.440 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: osdmap e53: 8 total, 7 up, 8 in 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: from='osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:16:01.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:01.441 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[40442]: 2024-04-06T03:16:01.135+0000 7f6da98b7700 -1 osd.7 0 waiting for initial osdmap 2024-04-06T03:16:01.441 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:16:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[40442]: 2024-04-06T03:16:01.143+0000 7f6da424e700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:16:02.403 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:02 smithi195 ceph-mon[27147]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:16:02.403 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:02 smithi195 ceph-mon[27147]: osdmap e54: 8 total, 7 up, 8 in 2024-04-06T03:16:02.403 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:02.403 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:02 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:02.438 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:02.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:02 smithi060 ceph-mon[24896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-06T03:16:02.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:02 smithi060 ceph-mon[24896]: osdmap e54: 8 total, 7 up, 8 in 2024-04-06T03:16:02.439 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:02.439 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:02 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:03.081 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1712373362,"num_in_osds":8,"osd_in_since":1712373344,"num_remapped_pgs":1} 2024-04-06T03:16:03.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:16:03.305 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:03.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: purged_snaps scrub starts 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: purged_snaps scrub ok 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708] boot 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: osdmap e55: 8 total, 8 up, 8 in 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:03.431 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:03 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2976165079' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T03:16:03.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: purged_snaps scrub starts 2024-04-06T03:16:03.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: purged_snaps scrub ok 2024-04-06T03:16:03.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:16:03.459 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: osd.7 [v2:172.21.15.195:6824/1286681708,v1:172.21.15.195:6825/1286681708] boot 2024-04-06T03:16:03.459 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: osdmap e55: 8 total, 8 up, 8 in 2024-04-06T03:16:03.459 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:16:03.459 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:03 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2976165079' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: osdmap e56: 8 total, 8 up, 8 in 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: Detected new or changed devices on smithi195 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi195 to 3984M 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:04.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: osdmap e56: 8 total, 8 up, 8 in 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: Detected new or changed devices on smithi195 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi195 to 3984M 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:04.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:16:05.225 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:05.225 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":58,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:09:57.255896+0000","modified":"2024-04-06T03:16:05.147623+0000","last_up_change":"2024-04-06T03:16:02.132805+0000","last_in_change":"2024-04-06T03:15:44.798194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:14:02.130810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"77456af4-610d-4353-8774-41203640a91d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6803","nonce":3771600438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6805","nonce":3771600438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6809","nonce":3771600438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6807","nonce":3771600438}]},"public_addr":"172.21.15.60:6803/3771600438","cluster_addr":"172.21.15.60:6805/3771600438","heartbeat_back_addr":"172.21.15.60:6809/3771600438","heartbeat_front_addr":"172.21.15.60:6807/3771600438","state":["exists","up"]},{"osd":1,"uuid":"99ecb76e-20b2-4e31-9d59-c3c5120183f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6811","nonce":3010589135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6813","nonce":3010589135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6817","nonce":3010589135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6815","nonce":3010589135}]},"public_addr":"172.21.15.60:6811/3010589135","cluster_addr":"172.21.15.60:6813/3010589135","heartbeat_back_addr":"172.21.15.60:6817/3010589135","heartbeat_front_addr":"172.21.15.60:6815/3010589135","state":["exists","up"]},{"osd":2,"uuid":"7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6819","nonce":1209852721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6821","nonce":1209852721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6825","nonce":1209852721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6823","nonce":1209852721}]},"public_addr":"172.21.15.60:6819/1209852721","cluster_addr":"172.21.15.60:6821/1209852721","heartbeat_back_addr":"172.21.15.60:6825/1209852721","heartbeat_front_addr":"172.21.15.60:6823/1209852721","state":["exists","up"]},{"osd":3,"uuid":"b68683e8-78ed-4864-bda4-461df17fa0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6827","nonce":2629454869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6829","nonce":2629454869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6833","nonce":2629454869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6831","nonce":2629454869}]},"public_addr":"172.21.15.60:6827/2629454869","cluster_addr":"172.21.15.60:6829/2629454869","heartbeat_back_addr":"172.21.15.60:6833/2629454869","heartbeat_front_addr":"172.21.15.60:6831/2629454869","state":["exists","up"]},{"osd":4,"uuid":"4fa47800-56f1-42f2-87cd-d16421d3ed08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6801","nonce":433073464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6803","nonce":433073464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6807","nonce":433073464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6805","nonce":433073464}]},"public_addr":"172.21.15.195:6801/433073464","cluster_addr":"172.21.15.195:6803/433073464","heartbeat_back_addr":"172.21.15.195:6807/433073464","heartbeat_front_addr":"172.21.15.195:6805/433073464","state":["exists","up"]},{"osd":5,"uuid":"a4ba76c8-b1e0-43ab-be8d-ce5d564573e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6809","nonce":1622341013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6811","nonce":1622341013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6815","nonce":1622341013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6813","nonce":1622341013}]},"public_addr":"172.21.15.195:6809/1622341013","cluster_addr":"172.21.15.195:6811/1622341013","heartbeat_back_addr":"172.21.15.195:6815/1622341013","heartbeat_front_addr":"172.21.15.195:6813/1622341013","state":["exists","up"]},{"osd":6,"uuid":"fbd4aaac-1089-4046-8880-4aa726906692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6817","nonce":2952424813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6819","nonce":2952424813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6823","nonce":2952424813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6821","nonce":2952424813}]},"public_addr":"172.21.15.195:6817/2952424813","cluster_addr":"172.21.15.195:6819/2952424813","heartbeat_back_addr":"172.21.15.195:6823/2952424813","heartbeat_front_addr":"172.21.15.195:6821/2952424813","state":["exists","up"]},{"osd":7,"uuid":"cd6732b5-ce77-48a8-80f1-688a0aa1044f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6825","nonce":1286681708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6827","nonce":1286681708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6831","nonce":1286681708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6829","nonce":1286681708}]},"public_addr":"172.21.15.195:6825/1286681708","cluster_addr":"172.21.15.195:6827/1286681708","heartbeat_back_addr":"172.21.15.195:6831/1286681708","heartbeat_front_addr":"172.21.15.195:6829/1286681708","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:12:59.507933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:30.667313+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:59.855417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:27.425460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:49.019915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:12.585400+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:36.252473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:16:00.189446+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2075859282":"2024-04-07T03:11:57.734144+0000","172.21.15.60:6801/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/2785742898":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/276471319":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4238723925":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6801/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/4063483083":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6800/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/3890322795":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/2626175489":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2444276683":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4170041718":"2024-04-07T03:10:33.955178+0000","172.21.15.60:0/828962699":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6801/2444276683":"2024-04-07T03:10:33.955178+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:16:05.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:05 smithi195 ceph-mon[27147]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:05.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:05 smithi195 ceph-mon[27147]: osdmap e57: 8 total, 8 up, 8 in 2024-04-06T03:16:05.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:05 smithi060 ceph-mon[24896]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:05.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:05 smithi060 ceph-mon[24896]: osdmap e57: 8 total, 8 up, 8 in 2024-04-06T03:16:05.995 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:14:02.130810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-06T03:16:05.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-06T03:16:06.220 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:06.454 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:06 smithi060 ceph-mon[24896]: osdmap e58: 8 total, 8 up, 8 in 2024-04-06T03:16:06.454 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:06 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/263201803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:06.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:06 smithi195 ceph-mon[27147]: osdmap e58: 8 total, 8 up, 8 in 2024-04-06T03:16:06.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:06 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/263201803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:07.371 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:07 smithi060 ceph-mon[24896]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:07.371 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:07 smithi060 ceph-mon[24896]: osdmap e59: 8 total, 8 up, 8 in 2024-04-06T03:16:07.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:07 smithi195 ceph-mon[27147]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:07.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:07 smithi195 ceph-mon[27147]: osdmap e59: 8 total, 8 up, 8 in 2024-04-06T03:16:08.079 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-06T03:16:08.379 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:08 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3799770095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T03:16:08.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:08 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3799770095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T03:16:08.805 INFO:tasks.cephadm:Setting up client nodes... 2024-04-06T03:16:08.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T03:16:09.433 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:09 smithi060 ceph-mon[24896]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:09.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:09 smithi195 ceph-mon[27147]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:10.986 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-04-06T03:16:10.987 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQB6vhBm8tANOxAAvNiYP/OuKGTSZ9ulXPRSww== 2024-04-06T03:16:11.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:11 smithi060 ceph-mon[24896]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T03:16:11.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:11 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2536511228' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:16:11.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:11 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2536511228' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:16:11.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:11 smithi195 ceph-mon[27147]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T03:16:11.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:11 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2536511228' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:16:11.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:11 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2536511228' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:16:11.676 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-06T03:16:11.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-06T03:16:11.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T03:16:11.718 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T03:16:11.718 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-06T03:16:11.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-06T03:16:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:13.436 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:13 smithi060 ceph-mon[24896]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-06T03:16:13.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:13 smithi195 ceph-mon[27147]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-06T03:16:13.941 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:14.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:14 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1995936830' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T03:16:14.428 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:14 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1995936830' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T03:16:14.661 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi060.jttwce","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1337711781},{"type":"v1","addr":"172.21.15.60:6801","nonce":1337711781}]},"active_addr":"172.21.15.60:6801/1337711781","active_change":"2024-04-06T03:11:57.734533+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi195.ultkot","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.60:8443/","prometheus":"http://172.21.15.60:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":525581087}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":830628189}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1812956136}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":4052587800}]}]}} 2024-04-06T03:16:14.666 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-06T03:16:14.666 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T03:16:14.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:16:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:15.320 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:15 smithi060 ceph-mon[24896]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-06T03:16:15.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:15 smithi195 ceph-mon[27147]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-06T03:16:16.600 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:16.601 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":59,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:09:57.255896+0000","modified":"2024-04-06T03:16:06.151864+0000","last_up_change":"2024-04-06T03:16:02.132805+0000","last_in_change":"2024-04-06T03:15:44.798194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:14:02.130810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"77456af4-610d-4353-8774-41203640a91d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6803","nonce":3771600438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6805","nonce":3771600438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6809","nonce":3771600438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6807","nonce":3771600438}]},"public_addr":"172.21.15.60:6803/3771600438","cluster_addr":"172.21.15.60:6805/3771600438","heartbeat_back_addr":"172.21.15.60:6809/3771600438","heartbeat_front_addr":"172.21.15.60:6807/3771600438","state":["exists","up"]},{"osd":1,"uuid":"99ecb76e-20b2-4e31-9d59-c3c5120183f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6811","nonce":3010589135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6813","nonce":3010589135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6817","nonce":3010589135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6815","nonce":3010589135}]},"public_addr":"172.21.15.60:6811/3010589135","cluster_addr":"172.21.15.60:6813/3010589135","heartbeat_back_addr":"172.21.15.60:6817/3010589135","heartbeat_front_addr":"172.21.15.60:6815/3010589135","state":["exists","up"]},{"osd":2,"uuid":"7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6819","nonce":1209852721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6821","nonce":1209852721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6825","nonce":1209852721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6823","nonce":1209852721}]},"public_addr":"172.21.15.60:6819/1209852721","cluster_addr":"172.21.15.60:6821/1209852721","heartbeat_back_addr":"172.21.15.60:6825/1209852721","heartbeat_front_addr":"172.21.15.60:6823/1209852721","state":["exists","up"]},{"osd":3,"uuid":"b68683e8-78ed-4864-bda4-461df17fa0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6827","nonce":2629454869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6829","nonce":2629454869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6833","nonce":2629454869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6831","nonce":2629454869}]},"public_addr":"172.21.15.60:6827/2629454869","cluster_addr":"172.21.15.60:6829/2629454869","heartbeat_back_addr":"172.21.15.60:6833/2629454869","heartbeat_front_addr":"172.21.15.60:6831/2629454869","state":["exists","up"]},{"osd":4,"uuid":"4fa47800-56f1-42f2-87cd-d16421d3ed08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6801","nonce":433073464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6803","nonce":433073464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6807","nonce":433073464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6805","nonce":433073464}]},"public_addr":"172.21.15.195:6801/433073464","cluster_addr":"172.21.15.195:6803/433073464","heartbeat_back_addr":"172.21.15.195:6807/433073464","heartbeat_front_addr":"172.21.15.195:6805/433073464","state":["exists","up"]},{"osd":5,"uuid":"a4ba76c8-b1e0-43ab-be8d-ce5d564573e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6809","nonce":1622341013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6811","nonce":1622341013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6815","nonce":1622341013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6813","nonce":1622341013}]},"public_addr":"172.21.15.195:6809/1622341013","cluster_addr":"172.21.15.195:6811/1622341013","heartbeat_back_addr":"172.21.15.195:6815/1622341013","heartbeat_front_addr":"172.21.15.195:6813/1622341013","state":["exists","up"]},{"osd":6,"uuid":"fbd4aaac-1089-4046-8880-4aa726906692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6817","nonce":2952424813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6819","nonce":2952424813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6823","nonce":2952424813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6821","nonce":2952424813}]},"public_addr":"172.21.15.195:6817/2952424813","cluster_addr":"172.21.15.195:6819/2952424813","heartbeat_back_addr":"172.21.15.195:6823/2952424813","heartbeat_front_addr":"172.21.15.195:6821/2952424813","state":["exists","up"]},{"osd":7,"uuid":"cd6732b5-ce77-48a8-80f1-688a0aa1044f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6825","nonce":1286681708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6827","nonce":1286681708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6831","nonce":1286681708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6829","nonce":1286681708}]},"public_addr":"172.21.15.195:6825/1286681708","cluster_addr":"172.21.15.195:6827/1286681708","heartbeat_back_addr":"172.21.15.195:6831/1286681708","heartbeat_front_addr":"172.21.15.195:6829/1286681708","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:12:59.507933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:30.667313+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:59.855417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:27.425460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:49.019915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:12.585400+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:36.252473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:16:00.189446+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2075859282":"2024-04-07T03:11:57.734144+0000","172.21.15.60:6801/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/2785742898":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/276471319":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4238723925":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6801/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/4063483083":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6800/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/3890322795":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/2626175489":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2444276683":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4170041718":"2024-04-07T03:10:33.955178+0000","172.21.15.60:0/828962699":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6801/2444276683":"2024-04-07T03:10:33.955178+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:16:17.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:17 smithi060 ceph-mon[24896]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T03:16:17.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:17 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3245414315' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:17.380 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T03:16:17.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:16:17.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:17 smithi195 ceph-mon[27147]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T03:16:17.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:17 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3245414315' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:19.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:19 smithi060 ceph-mon[24896]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-06T03:16:19.363 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:19.363 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":59,"fsid":"0ee7dd52-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:09:57.255896+0000","modified":"2024-04-06T03:16:06.151864+0000","last_up_change":"2024-04-06T03:16:02.132805+0000","last_in_change":"2024-04-06T03:15:44.798194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:14:02.130810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"77456af4-610d-4353-8774-41203640a91d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6803","nonce":3771600438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6805","nonce":3771600438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6809","nonce":3771600438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3771600438},{"type":"v1","addr":"172.21.15.60:6807","nonce":3771600438}]},"public_addr":"172.21.15.60:6803/3771600438","cluster_addr":"172.21.15.60:6805/3771600438","heartbeat_back_addr":"172.21.15.60:6809/3771600438","heartbeat_front_addr":"172.21.15.60:6807/3771600438","state":["exists","up"]},{"osd":1,"uuid":"99ecb76e-20b2-4e31-9d59-c3c5120183f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6811","nonce":3010589135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6813","nonce":3010589135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6817","nonce":3010589135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3010589135},{"type":"v1","addr":"172.21.15.60:6815","nonce":3010589135}]},"public_addr":"172.21.15.60:6811/3010589135","cluster_addr":"172.21.15.60:6813/3010589135","heartbeat_back_addr":"172.21.15.60:6817/3010589135","heartbeat_front_addr":"172.21.15.60:6815/3010589135","state":["exists","up"]},{"osd":2,"uuid":"7fb224ec-e2f3-4624-9c93-0e6d4a3bfa6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6819","nonce":1209852721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6821","nonce":1209852721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6825","nonce":1209852721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1209852721},{"type":"v1","addr":"172.21.15.60:6823","nonce":1209852721}]},"public_addr":"172.21.15.60:6819/1209852721","cluster_addr":"172.21.15.60:6821/1209852721","heartbeat_back_addr":"172.21.15.60:6825/1209852721","heartbeat_front_addr":"172.21.15.60:6823/1209852721","state":["exists","up"]},{"osd":3,"uuid":"b68683e8-78ed-4864-bda4-461df17fa0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6827","nonce":2629454869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6829","nonce":2629454869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6833","nonce":2629454869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2629454869},{"type":"v1","addr":"172.21.15.60:6831","nonce":2629454869}]},"public_addr":"172.21.15.60:6827/2629454869","cluster_addr":"172.21.15.60:6829/2629454869","heartbeat_back_addr":"172.21.15.60:6833/2629454869","heartbeat_front_addr":"172.21.15.60:6831/2629454869","state":["exists","up"]},{"osd":4,"uuid":"4fa47800-56f1-42f2-87cd-d16421d3ed08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6801","nonce":433073464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6803","nonce":433073464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6807","nonce":433073464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":433073464},{"type":"v1","addr":"172.21.15.195:6805","nonce":433073464}]},"public_addr":"172.21.15.195:6801/433073464","cluster_addr":"172.21.15.195:6803/433073464","heartbeat_back_addr":"172.21.15.195:6807/433073464","heartbeat_front_addr":"172.21.15.195:6805/433073464","state":["exists","up"]},{"osd":5,"uuid":"a4ba76c8-b1e0-43ab-be8d-ce5d564573e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6809","nonce":1622341013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6811","nonce":1622341013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6815","nonce":1622341013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1622341013},{"type":"v1","addr":"172.21.15.195:6813","nonce":1622341013}]},"public_addr":"172.21.15.195:6809/1622341013","cluster_addr":"172.21.15.195:6811/1622341013","heartbeat_back_addr":"172.21.15.195:6815/1622341013","heartbeat_front_addr":"172.21.15.195:6813/1622341013","state":["exists","up"]},{"osd":6,"uuid":"fbd4aaac-1089-4046-8880-4aa726906692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6817","nonce":2952424813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6819","nonce":2952424813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6823","nonce":2952424813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2952424813},{"type":"v1","addr":"172.21.15.195:6821","nonce":2952424813}]},"public_addr":"172.21.15.195:6817/2952424813","cluster_addr":"172.21.15.195:6819/2952424813","heartbeat_back_addr":"172.21.15.195:6823/2952424813","heartbeat_front_addr":"172.21.15.195:6821/2952424813","state":["exists","up"]},{"osd":7,"uuid":"cd6732b5-ce77-48a8-80f1-688a0aa1044f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6825","nonce":1286681708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6827","nonce":1286681708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6831","nonce":1286681708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1286681708},{"type":"v1","addr":"172.21.15.195:6829","nonce":1286681708}]},"public_addr":"172.21.15.195:6825/1286681708","cluster_addr":"172.21.15.195:6827/1286681708","heartbeat_back_addr":"172.21.15.195:6831/1286681708","heartbeat_front_addr":"172.21.15.195:6829/1286681708","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:12:59.507933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:30.667313+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:13:59.855417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:27.425460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:14:49.019915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:12.585400+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:15:36.252473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:16:00.189446+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2075859282":"2024-04-07T03:11:57.734144+0000","172.21.15.60:6801/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/2785742898":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/276471319":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4238723925":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6801/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/4063483083":"2024-04-07T03:11:03.775225+0000","172.21.15.60:6800/1179142736":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/3890322795":"2024-04-07T03:11:03.775225+0000","172.21.15.60:0/2626175489":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2444276683":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6800/2552645451":"2024-04-07T03:11:57.734144+0000","172.21.15.60:0/4170041718":"2024-04-07T03:10:33.955178+0000","172.21.15.60:0/828962699":"2024-04-07T03:10:33.955178+0000","172.21.15.60:6801/2444276683":"2024-04-07T03:10:33.955178+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:16:19.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:19 smithi195 ceph-mon[27147]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-06T03:16:20.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T03:16:20.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T03:16:20.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T03:16:20.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T03:16:20.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T03:16:20.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T03:16:20.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-06T03:16:20.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-06T03:16:20.465 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:20 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/705458071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:20.466 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.467 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.468 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.468 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.473 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.486 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:20.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:20 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/705458071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:16:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:21.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:21 smithi195 ceph-mon[27147]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-06T03:16:21.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:21 smithi060 ceph-mon[24896]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-06T03:16:23.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:23 smithi195 ceph-mon[27147]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:23.466 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:23 smithi060 ceph-mon[24896]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:25.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:25 smithi060 ceph-mon[24896]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:25.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:25 smithi195 ceph-mon[27147]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:27.347 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:27 smithi060 ceph-mon[24896]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:27.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:27 smithi195 ceph-mon[27147]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:29.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:29 smithi195 ceph-mon[27147]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:29.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:29 smithi060 ceph-mon[24896]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:31.057 INFO:teuthology.orchestra.run.smithi060.stdout:81604378657 2024-04-06T03:16:31.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T03:16:31.386 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:31 smithi060 ceph-mon[24896]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:31.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:31 smithi195 ceph-mon[27147]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:33.118 INFO:teuthology.orchestra.run.smithi060.stdout:111669149724 2024-04-06T03:16:33.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T03:16:33.343 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:33 smithi060 ceph-mon[24896]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:33.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:33 smithi195 ceph-mon[27147]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:34.448 INFO:teuthology.orchestra.run.smithi060.stdout:201863462926 2024-04-06T03:16:34.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-06T03:16:35.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:35 smithi060 ceph-mon[24896]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:35.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:35 smithi195 ceph-mon[27147]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:35.513 INFO:teuthology.orchestra.run.smithi060.stdout:38654705710 2024-04-06T03:16:35.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T03:16:35.646 INFO:teuthology.orchestra.run.smithi060.stdout:133143986199 2024-04-06T03:16:35.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T03:16:36.973 INFO:teuthology.orchestra.run.smithi060.stdout:236223201289 2024-04-06T03:16:36.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-06T03:16:37.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:37 smithi195 ceph-mon[27147]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:37.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:37 smithi060 ceph-mon[24896]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:37.546 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:37.546 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:37.962 INFO:teuthology.orchestra.run.smithi060.stdout:167503724564 2024-04-06T03:16:37.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T03:16:38.175 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:39.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:39 smithi195 ceph-mon[27147]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:39.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:39 smithi060 ceph-mon[24896]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:39.738 INFO:teuthology.orchestra.run.smithi060.stdout:60129542184 2024-04-06T03:16:39.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T03:16:41.339 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:41 smithi060 ceph-mon[24896]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:41.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:41 smithi195 ceph-mon[27147]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:43.418 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:43 smithi060 ceph-mon[24896]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:43.421 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:43.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:43 smithi195 ceph-mon[27147]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:43.626 INFO:teuthology.orchestra.run.smithi060.stdout:236223201290 2024-04-06T03:16:43.828 INFO:teuthology.orchestra.run.smithi060.stdout:201863462927 2024-04-06T03:16:44.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:44 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/518778946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:16:44.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:44 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/639244053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:16:44.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:44 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/518778946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:16:44.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:44 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/639244053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:16:45.268 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462926 got 201863462927 for osd.6 2024-04-06T03:16:45.269 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:45.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:45 smithi195 ceph-mon[27147]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:45.480 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:45 smithi060 ceph-mon[24896]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:46.212 INFO:teuthology.orchestra.run.smithi060.stdout:38654705712 2024-04-06T03:16:47.130 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:46 smithi060 ceph-mon[24896]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:47.131 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:46 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/581666997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:16:47.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:46 smithi195 ceph-mon[27147]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:47.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:46 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/581666997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:16:48.073 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201289 got 236223201290 for osd.7 2024-04-06T03:16:48.073 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:48.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705710 got 38654705712 for osd.0 2024-04-06T03:16:48.693 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:48.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:48 smithi060 ceph-mon[24896]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:49.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:48 smithi195 ceph-mon[27147]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:50.952 INFO:teuthology.orchestra.run.smithi060.stdout:81604378661 2024-04-06T03:16:51.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:50 smithi195 ceph-mon[27147]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:51.242 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:50 smithi060 ceph-mon[24896]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:52.099 INFO:teuthology.orchestra.run.smithi060.stdout:133143986204 2024-04-06T03:16:52.099 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:51 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/58289074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:16:52.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:51 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/58289074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:16:52.417 INFO:teuthology.orchestra.run.smithi060.stdout:167503724567 2024-04-06T03:16:52.814 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986199 got 133143986204 for osd.4 2024-04-06T03:16:52.814 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:53.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:52 smithi195 ceph-mon[27147]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:53.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:52 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2367016836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:16:53.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:52 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2763990329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:16:53.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:52 smithi060 ceph-mon[24896]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:53.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:52 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2367016836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:16:53.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:52 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2763990329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:16:53.641 INFO:teuthology.orchestra.run.smithi060.stdout:111669149728 2024-04-06T03:16:53.965 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:53 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1234914927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:16:54.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:53 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1234914927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:16:54.307 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149724 got 111669149728 for osd.3 2024-04-06T03:16:54.307 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:54.901 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378661 for osd.2 2024-04-06T03:16:54.901 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:55.140 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:54 smithi060 ceph-mon[24896]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:55.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:54 smithi195 ceph-mon[27147]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:55.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724564 got 167503724567 for osd.5 2024-04-06T03:16:55.513 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:55.894 INFO:teuthology.orchestra.run.smithi060.stdout:60129542188 2024-04-06T03:16:56.511 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542184 got 60129542188 for osd.1 2024-04-06T03:16:56.511 DEBUG:teuthology.parallel:result is None 2024-04-06T03:16:56.511 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T03:16:56.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:16:56.736 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:16:57.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:56 smithi060 ceph-mon[24896]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:57.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:56 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3821290882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:16:57.215 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:56 smithi195 ceph-mon[27147]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:57.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:56 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3821290882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:16:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:16:58.468 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-06T03:16:59.065 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:58 smithi060 ceph-mon[24896]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:59.066 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:16:59.066 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:16:59.066 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:16:58 smithi060 ceph-mon[24896]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:16:59.091 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-06T03:16:57.858655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48880,"kb_used_data":6320,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912336,"statfs":{"total":767960285184,"available":767910232064,"internally_reserved":0,"allocated":6471680,"data_stored":3548060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002100"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-06T03:16:06.160011+0000","last_change":"2024-04-06T03:16:06.160011+0000","last_active":"2024-04-06T03:16:06.160011+0000","last_peered":"2024-04-06T03:16:06.160011+0000","last_clean":"2024-04-06T03:16:06.160011+0000","last_became_active":"2024-04-06T03:16:06.159520+0000","last_became_peered":"2024-04-06T03:16:06.159520+0000","last_unstale":"2024-04-06T03:16:06.160011+0000","last_undegraded":"2024-04-06T03:16:06.160011+0000","last_fullsized":"2024-04-06T03:16:06.160011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:14:02.745466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:14:02.745466+0000","last_clean_scrub_stamp":"2024-04-06T03:14:02.745466+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:17:01.767351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.435}]},{"osd":4,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.499},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.246,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.35599999999999998,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Apr 6 03:16:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.379,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.379},"min":{"1min":0.29799999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 6 03:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.314},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.32400000000000001}]},{"osd":4,"last update":"Sat Apr 6 03:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 03:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 03:16:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.32300000000000001},"min":{"1min":0.23499999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.54700000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.32300000000000001},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 6 03:16:04 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.36399999999999999,"15min":0.35199999999999998},"min":{"1min":0.28599999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40000000000000002,"15min":0.39000000000000001},"min":{"1min":0.377,"5min":0.252,"15min":0.252},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Apr 6 03:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Apr 6 03:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Apr 6 03:16:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":12.006,"5min":12.006,"15min":12.006},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Sat Apr 6 03:16:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.27800000000000002,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.317}]},{"osd":1,"last update":"Sat Apr 6 03:16:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.29599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.24099999999999999}]},{"osd":3,"last update":"Sat Apr 6 03:16:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.26000000000000001}]},{"osd":4,"last update":"Sat Apr 6 03:16:56 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.57999999999999996},"min":{"1min":0.503,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.622,"15min":0.62},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Apr 6 03:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.496}]},{"osd":1,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.432,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.56599999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48099999999999998,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.45900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.623}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.70099999999999996}]},{"osd":6,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44}]},{"osd":3,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41999999999999998}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63900000000000001}]},{"osd":1,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.495},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.40000000000000002}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:16:59.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:16:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:58 smithi195 ceph-mon[27147]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:16:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:16:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:16:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:16:58 smithi195 ceph-mon[27147]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:16:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:17:00.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:00 smithi060 ceph-mon[24896]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:01.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:01.038 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-06T03:17:01.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:00 smithi195 ceph-mon[27147]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:01.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-06T03:16:59.858926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48880,"kb_used_data":6320,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912336,"statfs":{"total":767960285184,"available":767910232064,"internally_reserved":0,"allocated":6471680,"data_stored":3548060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001901"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-06T03:16:06.160011+0000","last_change":"2024-04-06T03:16:06.160011+0000","last_active":"2024-04-06T03:16:06.160011+0000","last_peered":"2024-04-06T03:16:06.160011+0000","last_clean":"2024-04-06T03:16:06.160011+0000","last_became_active":"2024-04-06T03:16:06.159520+0000","last_became_peered":"2024-04-06T03:16:06.159520+0000","last_unstale":"2024-04-06T03:16:06.160011+0000","last_undegraded":"2024-04-06T03:16:06.160011+0000","last_fullsized":"2024-04-06T03:16:06.160011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:14:02.745466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:14:02.745466+0000","last_clean_scrub_stamp":"2024-04-06T03:14:02.745466+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:17:01.767351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.5},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.437}]},{"osd":1,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.47599999999999998}]},{"osd":3,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41899999999999998}]},{"osd":4,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.246,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.35599999999999998,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 6 03:16:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.379,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.315},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.379},"min":{"1min":0.29799999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.433}]},{"osd":3,"last update":"Sat Apr 6 03:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.375},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Apr 6 03:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.83899999999999997}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.82799999999999996}]},{"osd":6,"last update":"Sat Apr 6 03:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 03:16:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.32300000000000001},"min":{"1min":0.23499999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.54700000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.32300000000000001},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 6 03:16:04 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.36399999999999999,"15min":0.35199999999999998},"min":{"1min":0.28599999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40000000000000002,"15min":0.39000000000000001},"min":{"1min":0.377,"5min":0.252,"15min":0.252},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Apr 6 03:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Apr 6 03:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Apr 6 03:16:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":12.006,"5min":12.006,"15min":12.006},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Sat Apr 6 03:16:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.27800000000000002,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.317}]},{"osd":1,"last update":"Sat Apr 6 03:16:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.29599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.24099999999999999}]},{"osd":3,"last update":"Sat Apr 6 03:16:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.26000000000000001}]},{"osd":4,"last update":"Sat Apr 6 03:16:56 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.57999999999999996},"min":{"1min":0.503,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.622,"15min":0.62},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Apr 6 03:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.254},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.311},{"interface":"front","average":{"1min":0.432,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.56599999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.41099999999999998}]},{"osd":2,"last update":"Sat Apr 6 03:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48099999999999998,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.45900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.371}]},{"osd":4,"last update":"Sat Apr 6 03:16:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Apr 6 03:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.621},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44}]},{"osd":3,"last update":"Sat Apr 6 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41999999999999998}]},{"osd":5,"last update":"Sat Apr 6 03:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63900000000000001}]},{"osd":1,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.495},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Sat Apr 6 03:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.40000000000000002}]},{"osd":6,"last update":"Sat Apr 6 03:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:17:01.671 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T03:17:01.671 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T03:17:01.672 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-06T03:17:01.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-06T03:17:01.893 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/mon.smithi060/config 2024-04-06T03:17:01.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:01 smithi060 ceph-mon[24896]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:02.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:01 smithi195 ceph-mon[27147]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:02.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:02 smithi060 ceph-mon[24896]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:03.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:02 smithi195 ceph-mon[27147]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:03.778 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T03:17:04.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:04.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:04.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:04.106 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:03 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2817908930' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T03:17:04.110 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:04.110 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:04.110 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:04.110 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:03 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2817908930' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T03:17:04.451 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-06T03:17:04.451 INFO:tasks.cephadm:Setup complete, yielding 2024-04-06T03:17:04.452 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T03:17:04.467 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:04.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-06T03:17:05.440 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:05 smithi060 ceph-mon[24896]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:05.440 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:05 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:05.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:05 smithi195 ceph-mon[27147]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:05.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:05 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:06.496 INFO:teuthology.orchestra.run.smithi060.stdout:Backend: cephadm 2024-04-06T03:17:06.496 INFO:teuthology.orchestra.run.smithi060.stdout:Available: Yes 2024-04-06T03:17:06.496 INFO:teuthology.orchestra.run.smithi060.stdout:Paused: No 2024-04-06T03:17:07.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-06T03:17:07.163 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:07 smithi060 ceph-mon[24896]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:07.163 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:07 smithi060 ceph-mon[24896]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:07.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:07 smithi195 ceph-mon[27147]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:07.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:07 smithi195 ceph-mon[27147]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:09.149 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 5a135361e3ed 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (5m) 2m ago 5m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (5m) 68s ago 5m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (3m) 2m ago 5m 45.6M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (6m) 2m ago 6m 449M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (4m) 68s ago 4m 414M - 17.2.0 e1d6a67b021e c9b1a21c5cf5 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (6m) 2m ago 7m 43.5M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (4m) 68s ago 4m 42.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (5m) 2m ago 5m 17.6M - 1dbe0e931976 2c4375f28350 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (4m) 68s ago 4m 18.3M - 1dbe0e931976 a0369313f174 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (4m) 2m ago 4m 34.2M 3217M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (3m) 2m ago 3m 37.4M 3217M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:17:09.150 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (3m) 2m ago 3m 33.1M 3217M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (2m) 2m ago 2m 25.5M 3217M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (2m) 68s ago 2m 35.7M 3984M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (2m) 68s ago 2m 35.5M 3984M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (97s) 68s ago 97s 30.4M 3984M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (73s) 68s ago 73s 12.0M 3984M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:17:09.151 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (4m) 2m ago 4m 53.6M - 514e6a882f6e 03d658f74f0b 2024-04-06T03:17:09.452 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:09 smithi060 ceph-mon[24896]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:09.465 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:09 smithi195 ceph-mon[27147]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:09.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-06T03:17:10.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:10 smithi060 ceph-mon[24896]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:10.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:10 smithi195 ceph-mon[27147]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:11.429 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:11 smithi060 ceph-mon[24896]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:11.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:11 smithi195 ceph-mon[27147]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:crash 2/2 2m ago 6m * 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:mgr 2/2 2m ago 6m count:2 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:mon 2/2 2m ago 5m smithi060:172.21.15.60=smithi060;smithi195:172.21.15.195=smithi195;count:2 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:osd 8 2m ago - 2024-04-06T03:17:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-06T03:17:12.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-06T03:17:13.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:13 smithi195 ceph-mon[27147]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:13.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:13 smithi195 ceph-mon[27147]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:13.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:13 smithi060 ceph-mon[24896]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:13.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:13 smithi060 ceph-mon[24896]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:14.522 INFO:teuthology.orchestra.run.smithi060.stdout:HOST ADDR LABELS STATUS 2024-04-06T03:17:14.522 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060 172.21.15.60 2024-04-06T03:17:14.522 INFO:teuthology.orchestra.run.smithi060.stdout:smithi195 172.21.15.195 2024-04-06T03:17:14.522 INFO:teuthology.orchestra.run.smithi060.stdout:2 hosts in cluster 2024-04-06T03:17:15.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-06T03:17:15.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:15 smithi195 ceph-mon[27147]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:15.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:15 smithi195 ceph-mon[27147]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:15.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:15 smithi060 ceph-mon[24896]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:15.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:15 smithi060 ceph-mon[24896]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:17.124 INFO:teuthology.orchestra.run.smithi060.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-06T03:17:17.125 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476008A480DGN 480G LVM detected, locked 2024-04-06T03:17:17.125 INFO:teuthology.orchestra.run.smithi060.stdout:smithi195 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300FS480DGN 480G LVM detected, locked 2024-04-06T03:17:17.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:17 smithi195 ceph-mon[27147]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:17.491 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:17 smithi060 ceph-mon[24896]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:17.726 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-06T03:17:17.740 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:17.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-06T03:17:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:+ systemctl stop nfs-server 2024-04-06T03:17:17.776 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi195.front.sepia.ceph.com 2024-04-06T03:17:17.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-06T03:17:17.809 INFO:teuthology.orchestra.run.smithi195.stderr:+ systemctl stop nfs-server 2024-04-06T03:17:17.817 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T03:17:17.830 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:17.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create foofs' 2024-04-06T03:17:18.387 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:18 smithi060 ceph-mon[24896]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:18.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:18 smithi195 ceph-mon[27147]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:19.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:19 smithi060 ceph-mon[24896]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:19.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:19 smithi195 ceph-mon[27147]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:20.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:20 smithi195 ceph-mon[27147]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:20.814 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:20 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-06T03:17:20.815 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:20 smithi060 ceph-mon[24896]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:20.815 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:20 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-06T03:17:21.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:21 smithi195 ceph-mon[27147]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:21.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:21 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-06T03:17:21.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:21 smithi195 ceph-mon[27147]: osdmap e60: 8 total, 8 up, 8 in 2024-04-06T03:17:21.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:21 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-06T03:17:21.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:21 smithi060 ceph-mon[24896]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:21.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:21 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-06T03:17:21.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:21 smithi060 ceph-mon[24896]: osdmap e60: 8 total, 8 up, 8 in 2024-04-06T03:17:21.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:21 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-06T03:17:21.479 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:21 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[24892]: 2024-04-06T03:17:21.165+0000 7f2831d7c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:17:21.852 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-06T03:17:21.861 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-06T03:17:21.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: osdmap e61: 8 total, 8 up, 8 in 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: osdmap e62: 8 total, 8 up, 8 in 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-06T03:17:22.390 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: fsmap foofs:0 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: Saving service mds.foofs spec with placement count:2 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:22.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:22 smithi060 ceph-mon[24896]: Deploying daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:17:22.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-06T03:17:22.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: osdmap e61: 8 total, 8 up, 8 in 2024-04-06T03:17:22.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: osdmap e62: 8 total, 8 up, 8 in 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: fsmap foofs:0 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: Saving service mds.foofs spec with placement count:2 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:17:22.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T03:17:22.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:22.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:22 smithi195 ceph-mon[27147]: Deploying daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:17:23.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:23 smithi195 ceph-mon[27147]: pgmap v212: 65 pgs: 11 creating+peering, 8 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:23.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:23 smithi195 ceph-mon[27147]: osdmap e63: 8 total, 8 up, 8 in 2024-04-06T03:17:23.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:23 smithi060 ceph-mon[24896]: pgmap v212: 65 pgs: 11 creating+peering, 8 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:23.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:23 smithi060 ceph-mon[24896]: osdmap e63: 8 total, 8 up, 8 in 2024-04-06T03:17:24.426 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:24 smithi060 ceph-mon[24896]: osdmap e64: 8 total, 8 up, 8 in 2024-04-06T03:17:24.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:24 smithi195 ceph-mon[27147]: osdmap e64: 8 total, 8 up, 8 in 2024-04-06T03:17:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:25.153 INFO:teuthology.orchestra.run.smithi060.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-06T03:10:53.238905Z", "last_refresh": "2024-04-06T03:14:37.128950Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-06T03:10:46.906353Z", "last_refresh": "2024-04-06T03:14:37.129037Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-06T03:10:49.934828Z", "last_refresh": "2024-04-06T03:14:37.129109Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-06T03:17:21.188597Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-06T03:17:21.184607Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-06T03:10:44.792800Z", "last_refresh": "2024-04-06T03:14:37.128874Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi060:172.21.15.60=smithi060", "smithi195:172.21.15.195=smithi195"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-06T03:11:37.279420Z", "last_refresh": "2024-04-06T03:14:37.128731Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-06T03:10:51.152531Z", "last_refresh": "2024-04-06T03:14:37.129174Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-06T03:14:37.129302Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-06T03:10:48.717338Z", "last_refresh": "2024-04-06T03:14:37.129238Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-06T03:17:25.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:25 smithi195 ceph-mon[27147]: pgmap v215: 65 pgs: 11 creating+peering, 40 active+clean, 14 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:25.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:25 smithi060 ceph-mon[24896]: pgmap v215: 65 pgs: 11 creating+peering, 40 active+clean, 14 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:25.916 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-06T03:17:26.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:26 smithi195 ceph-mon[27147]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:26.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:26 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:26.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:26 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:17:26.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:26 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T03:17:26.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:26 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:26.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:26 smithi060 ceph-mon[24896]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:26.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:26 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:26.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:26 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:17:26.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:26 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T03:17:26.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:26 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:26.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: pgmap v216: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: Deploying daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: mds.? [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] up:boot 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: daemon mds.foofs.smithi060.hrodqu assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: Cluster is now healthy 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: fsmap foofs:0 1 up:standby 2024-04-06T03:17:27.221 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:17:27.222 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:creating} 2024-04-06T03:17:27.222 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:27 smithi060 ceph-mon[24896]: daemon mds.foofs.smithi060.hrodqu is now active in filesystem foofs as rank 0 2024-04-06T03:17:27.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: pgmap v216: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: Deploying daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: mds.? [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] up:boot 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: daemon mds.foofs.smithi060.hrodqu assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: Cluster is now healthy 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: fsmap foofs:0 1 up:standby 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:creating} 2024-04-06T03:17:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:27 smithi195 ceph-mon[27147]: daemon mds.foofs.smithi060.hrodqu is now active in filesystem foofs as rank 0 2024-04-06T03:17:29.056 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:29.056 INFO:teuthology.orchestra.run.smithi060.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-06T03:10:53.238905Z", "last_refresh": "2024-04-06T03:14:37.128950Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-06T03:10:46.906353Z", "last_refresh": "2024-04-06T03:14:37.129037Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-06T03:10:49.934828Z", "last_refresh": "2024-04-06T03:14:37.129109Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-06T03:17:21.188597Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-06T03:17:21.184607Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-06T03:10:44.792800Z", "last_refresh": "2024-04-06T03:14:37.128874Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi060:172.21.15.60=smithi060", "smithi195:172.21.15.195=smithi195"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-06T03:11:37.279420Z", "last_refresh": "2024-04-06T03:14:37.128731Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-06T03:10:51.152531Z", "last_refresh": "2024-04-06T03:14:37.129174Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-06T03:14:37.129302Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-06T03:10:48.717338Z", "last_refresh": "2024-04-06T03:14:37.129238Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: mds.? [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] up:active 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:29.189 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:28 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: mds.? [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] up:active 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:29.193 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:28 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:30.319 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-06T03:17:30.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:30 smithi195 ceph-mon[27147]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:30.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:30 smithi195 ceph-mon[27147]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:boot 2024-04-06T03:17:30.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:30 smithi195 ceph-mon[27147]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:17:30.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:30 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:17:30.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:30 smithi195 ceph-mon[27147]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:17:30.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:30 smithi060 ceph-mon[24896]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:30.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:30 smithi060 ceph-mon[24896]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:boot 2024-04-06T03:17:30.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:30 smithi060 ceph-mon[24896]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:17:30.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:30 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:17:30.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:30 smithi060 ceph-mon[24896]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:17:31.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-06T03:17:31.873 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:31 smithi060 ceph-mon[24896]: pgmap v218: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 4 op/s 2024-04-06T03:17:31.874 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:31 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:31.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:31 smithi195 ceph-mon[27147]: pgmap v218: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 4 op/s 2024-04-06T03:17:31.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:31 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:33.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:33 smithi195 ceph-mon[27147]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-06T03:17:33.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:33 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:33.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:33 smithi060 ceph-mon[24896]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-06T03:17:33.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:33 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:34.978 INFO:teuthology.orchestra.run.smithi060.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-06T03:10:53.238905Z", "last_refresh": "2024-04-06T03:17:34.943322Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-06T03:10:46.906353Z", "last_refresh": "2024-04-06T03:17:30.657463Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-06T03:10:49.934828Z", "last_refresh": "2024-04-06T03:17:34.943452Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-06T03:17:21.188597Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-06T03:17:21.184607Z", "last_refresh": "2024-04-06T03:17:30.658076Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-06T03:10:44.792800Z", "last_refresh": "2024-04-06T03:17:30.657609Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi060:172.21.15.60=smithi060", "smithi195:172.21.15.195=smithi195"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-06T03:11:37.279420Z", "last_refresh": "2024-04-06T03:17:30.657684Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-06T03:10:51.152531Z", "last_refresh": "2024-04-06T03:17:30.657750Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-06T03:17:30.657813Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-06T03:10:48.717338Z", "last_refresh": "2024-04-06T03:17:34.943577Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-06T03:17:35.893 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-06T03:17:35.893 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T03:17:35.903 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:35.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-06T03:17:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-06T03:17:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:35.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:35.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:35.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:35.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:35 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:35.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-06T03:17:35.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:35.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:35.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:35.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:35.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:35.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:35 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:36.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:36 smithi060 ceph-mon[24896]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:37.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:36 smithi195 ceph-mon[27147]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:17:37.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:37 smithi060 ceph-mon[24896]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-06T03:17:38.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:37 smithi195 ceph-mon[27147]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-06T03:17:38.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:38 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-06T03:17:39.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:38 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-06T03:17:39.766 INFO:teuthology.orchestra.run.smithi060.stdout:NFS Cluster Created Successfully 2024-04-06T03:17:39.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:39 smithi060 ceph-mon[24896]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-06T03:17:39.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:39 smithi060 ceph-mon[24896]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:39.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:39 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-06T03:17:39.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:39 smithi060 ceph-mon[24896]: osdmap e65: 8 total, 8 up, 8 in 2024-04-06T03:17:39.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:39 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-06T03:17:40.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:39 smithi195 ceph-mon[27147]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-06T03:17:40.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:39 smithi195 ceph-mon[27147]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:40.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:39 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-06T03:17:40.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:39 smithi195 ceph-mon[27147]: osdmap e65: 8 total, 8 up, 8 in 2024-04-06T03:17:40.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:39 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-06T03:17:40.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-06T03:17:40.894 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-06T03:17:40.894 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: osdmap e66: 8 total, 8 up, 8 in 2024-04-06T03:17:40.894 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: Saving service nfs.foo spec with placement count:2 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: Creating key for client.nfs.foo.0.0.smithi060.natxue 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:40.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:17:40.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:17:40.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:17:40.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:17:40.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:40 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: osdmap e66: 8 total, 8 up, 8 in 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: Saving service nfs.foo spec with placement count:2 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: Creating key for client.nfs.foo.0.0.smithi060.natxue 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:17:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:17:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:40 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:41.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:41 smithi060 ceph-mon[24896]: pgmap v225: 97 pgs: 4 creating+activating, 25 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:41.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:41 smithi060 ceph-mon[24896]: Rados config object exists: conf-nfs.foo 2024-04-06T03:17:41.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:41 smithi060 ceph-mon[24896]: Creating key for client.nfs.foo.0.0.smithi060.natxue-rgw 2024-04-06T03:17:41.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:41 smithi060 ceph-mon[24896]: Deploying daemon nfs.foo.0.0.smithi060.natxue on smithi060 2024-04-06T03:17:41.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:41 smithi060 ceph-mon[24896]: osdmap e67: 8 total, 8 up, 8 in 2024-04-06T03:17:42.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:41 smithi195 ceph-mon[27147]: pgmap v225: 97 pgs: 4 creating+activating, 25 unknown, 68 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:42.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:41 smithi195 ceph-mon[27147]: Rados config object exists: conf-nfs.foo 2024-04-06T03:17:42.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:41 smithi195 ceph-mon[27147]: Creating key for client.nfs.foo.0.0.smithi060.natxue-rgw 2024-04-06T03:17:42.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:41 smithi195 ceph-mon[27147]: Deploying daemon nfs.foo.0.0.smithi060.natxue on smithi060 2024-04-06T03:17:42.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:41 smithi195 ceph-mon[27147]: osdmap e67: 8 total, 8 up, 8 in 2024-04-06T03:17:43.118 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:42 smithi195 ceph-mon[27147]: pgmap v227: 97 pgs: 4 creating+activating, 9 unknown, 84 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:43.119 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:42 smithi195 ceph-mon[27147]: mgrmap e20: smithi060.jttwce(active, since 5m), standbys: smithi195.ultkot 2024-04-06T03:17:43.130 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:42 smithi060 ceph-mon[24896]: pgmap v227: 97 pgs: 4 creating+activating, 9 unknown, 84 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:43.130 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:42 smithi060 ceph-mon[24896]: mgrmap e20: smithi060.jttwce(active, since 5m), standbys: smithi195.ultkot 2024-04-06T03:17:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-06T03:17:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-06T03:17:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:Error EINVAL: invalid command 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "bind": "/fake", 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "fs": "foofs", 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster": "foo", 2024-04-06T03:17:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": "RW" 2024-04-06T03:17:44.038 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:17:44.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-06T03:17:45.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: pgmap v228: 97 pgs: 4 creating+activating, 93 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:45.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:45.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:45.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy 2024-04-06T03:17:45.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-06T03:17:45.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:45 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:45.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: pgmap v228: 97 pgs: 4 creating+activating, 93 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-06T03:17:45.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:45 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:17:47.243 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:47 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:17:47.366 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:47 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:48.416 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:48 smithi195 ceph-mon[27147]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:48.416 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:48 smithi195 ceph-mon[27147]: Rados config object exists: conf-nfs.foo 2024-04-06T03:17:48.416 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:48 smithi195 ceph-mon[27147]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy-rgw 2024-04-06T03:17:48.416 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:48 smithi195 ceph-mon[27147]: Deploying daemon nfs.foo.1.0.smithi195.rdvmyy on smithi195 2024-04-06T03:17:48.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:48 smithi060 ceph-mon[24896]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:48.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:48 smithi060 ceph-mon[24896]: Rados config object exists: conf-nfs.foo 2024-04-06T03:17:48.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:48 smithi060 ceph-mon[24896]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy-rgw 2024-04-06T03:17:48.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:48 smithi060 ceph-mon[24896]: Deploying daemon nfs.foo.1.0.smithi195.rdvmyy on smithi195 2024-04-06T03:17:49.108 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:49 smithi195 ceph-mon[27147]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-06T03:17:49.109 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:49 smithi195 ceph-mon[27147]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:49.311 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:49 smithi060 ceph-mon[24896]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-06T03:17:49.311 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:49 smithi060 ceph-mon[24896]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:50.606 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:50.606 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:50.606 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:50.607 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:50 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:50.607 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:50 smithi060 ceph-mon[24896]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:50.613 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:50.614 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:50.614 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:50.614 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:50 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:50.614 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:50 smithi195 ceph-mon[27147]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:51.585 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:51 smithi060 ceph-mon[24896]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-06T03:17:51.635 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:51 smithi195 ceph-mon[27147]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-06T03:17:52.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:52 smithi060 ceph-mon[24896]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:52.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:52 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:52.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:52 smithi195 ceph-mon[27147]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:52.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:52 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:53.938 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:53 smithi060 ceph-mon[24896]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-06T03:17:53.938 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:53 smithi060 ceph-mon[24896]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:53.938 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:53 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:53 smithi195 ceph-mon[27147]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-06T03:17:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:53 smithi195 ceph-mon[27147]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:53 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.825 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo ?:2049 2/2 2s ago 15s count:2 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:54.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:54 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:54 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:17:55.574 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-06T03:17:55.584 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:55.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-06T03:17:55.612 INFO:teuthology.orchestra.run.smithi060.stderr:+ mkdir /mnt/foo 2024-04-06T03:17:55.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-06T03:17:55.706 INFO:teuthology.orchestra.run.smithi060.stderr:++ hostname 2024-04-06T03:17:55.710 INFO:teuthology.orchestra.run.smithi060.stderr:+ mount -t nfs smithi060:/fake /mnt/foo -o sync 2024-04-06T03:17:55.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:55 smithi195 ceph-mon[27147]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-06T03:17:55.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:55 smithi195 ceph-mon[27147]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:55.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:55 smithi060 ceph-mon[24896]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-06T03:17:55.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:55 smithi060 ceph-mon[24896]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:56.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-06T03:17:56.191 INFO:teuthology.orchestra.run.smithi060.stderr:+ echo test 2024-04-06T03:17:56.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-06T03:17:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:+ sync 2024-04-06T03:17:56.416 INFO:teuthology.run_tasks:Running task parallel... 2024-04-06T03:17:56.434 INFO:teuthology.task.parallel:starting parallel... 2024-04-06T03:17:56.434 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T03:17:56.435 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T03:17:56.435 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:56.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-06T03:17:56.436 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T03:17:56.436 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-06T03:17:56.437 INFO:teuthology.task.exec:Executing custom commands... 2024-04-06T03:17:56.437 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:17:56.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-06T03:17:56.495 INFO:teuthology.orchestra.run.smithi060.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-06T03:17:56.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:17:56.495 INFO:teuthology.orchestra.run.smithi060.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-06T03:17:56.496 INFO:teuthology.orchestra.run.smithi060.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-06T03:17:56.597 INFO:teuthology.orchestra.run.smithi060.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-06T03:17:56.597 INFO:teuthology.orchestra.run.smithi060.stdout:releasing clients 2024-04-06T03:17:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 138 30.12 MB/sec warmup 1 sec latency 31.905 ms 2024-04-06T03:17:57.942 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:57 smithi060 ceph-mon[24896]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-06T03:17:58.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:57 smithi195 ceph-mon[27147]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-06T03:17:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 280 26.05 MB/sec warmup 2 sec latency 31.316 ms 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:17:58.993 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:17:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-04-06T03:17:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-04-06T03:17:59.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-04-06T03:17:59.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-06T03:17:59.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:17:59.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:17:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 406 23.98 MB/sec warmup 3 sec latency 29.660 ms 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: pgmap v235: 97 pgs: 97 active+clean; 1.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 87 KiB/s wr, 5 op/s 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-06T03:17:59.790 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:17:59 smithi060 ceph-mon[24896]: osdmap e68: 8 total, 8 up, 8 in 2024-04-06T03:18:00.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-06T03:18:00.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: pgmap v235: 97 pgs: 97 active+clean; 1.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 87 KiB/s wr, 5 op/s 2024-04-06T03:18:00.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-04-06T03:18:00.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-04-06T03:18:00.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-04-06T03:18:00.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-06T03:18:00.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:17:59 smithi195 ceph-mon[27147]: osdmap e68: 8 total, 8 up, 8 in 2024-04-06T03:18:00.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 546 23.57 MB/sec warmup 4 sec latency 21.827 ms 2024-04-06T03:18:00.909 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:00 smithi060 ceph-mon[24896]: osdmap e69: 8 total, 8 up, 8 in 2024-04-06T03:18:01.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:00 smithi195 ceph-mon[27147]: osdmap e69: 8 total, 8 up, 8 in 2024-04-06T03:18:01.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 792 21.43 MB/sec warmup 5 sec latency 46.580 ms 2024-04-06T03:18:01.918 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:01 smithi060 ceph-mon[24896]: pgmap v238: 97 pgs: 97 active+clean; 37 MiB data, 82 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 4.7 MiB/s wr, 201 op/s 2024-04-06T03:18:02.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:01 smithi195 ceph-mon[27147]: pgmap v238: 97 pgs: 97 active+clean; 37 MiB data, 82 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 4.7 MiB/s wr, 201 op/s 2024-04-06T03:18:02.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 1189 19.20 MB/sec warmup 6 sec latency 38.681 ms 2024-04-06T03:18:03.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 2144 19.08 MB/sec warmup 7 sec latency 15.977 ms 2024-04-06T03:18:03.941 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:03 smithi060 ceph-mon[24896]: pgmap v239: 97 pgs: 97 active+clean; 80 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 10 MiB/s wr, 309 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:04.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:03 smithi195 ceph-mon[27147]: pgmap v239: 97 pgs: 97 active+clean; 80 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 10 MiB/s wr, 309 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:04.597 INFO:teuthology.orchestra.run.smithi060.stdout: 5 2965 20.94 MB/sec warmup 8 sec latency 18.968 ms 2024-04-06T03:18:04.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-06T03:18:05.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 3736 21.70 MB/sec warmup 9 sec latency 45.839 ms 2024-04-06T03:18:05.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:05 smithi195 ceph-mon[27147]: pgmap v240: 97 pgs: 97 active+clean; 103 MiB data, 174 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 13 MiB/s wr, 512 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:05.990 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:05 smithi060 ceph-mon[24896]: pgmap v240: 97 pgs: 97 active+clean; 103 MiB data, 174 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 13 MiB/s wr, 512 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:06.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 4019 20.03 MB/sec warmup 10 sec latency 32.463 ms 2024-04-06T03:18:07.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 4715 19.14 MB/sec warmup 11 sec latency 21.835 ms 2024-04-06T03:18:07.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:07 smithi060 ceph-mon[24896]: pgmap v241: 97 pgs: 97 active+clean; 115 MiB data, 198 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 15 MiB/s wr, 731 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:08.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:07 smithi195 ceph-mon[27147]: pgmap v241: 97 pgs: 97 active+clean; 115 MiB data, 198 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 15 MiB/s wr, 731 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:08.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 5618 19.06 MB/sec warmup 12 sec latency 16.025 ms 2024-04-06T03:18:09.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 6373 19.71 MB/sec warmup 13 sec latency 24.706 ms 2024-04-06T03:18:09.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph mgr module enable nfs --force' 2024-04-06T03:18:09.772 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:09 smithi060 ceph-mon[24896]: pgmap v242: 97 pgs: 97 active+clean; 124 MiB data, 227 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 13 MiB/s wr, 677 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:10.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:09 smithi195 ceph-mon[27147]: pgmap v242: 97 pgs: 97 active+clean; 124 MiB data, 227 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 13 MiB/s wr, 677 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:10.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 6992 19.91 MB/sec warmup 14 sec latency 42.195 ms 2024-04-06T03:18:11.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 7450 19.71 MB/sec warmup 15 sec latency 32.488 ms 2024-04-06T03:18:11.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:11 smithi060 ceph-mon[24896]: pgmap v243: 97 pgs: 97 active+clean; 139 MiB data, 267 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 989 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:12.216 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:11 smithi195 ceph-mon[27147]: pgmap v243: 97 pgs: 97 active+clean; 139 MiB data, 267 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 989 op/s; 2 B/s, 0 objects/s recovering 2024-04-06T03:18:12.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 7993 18.71 MB/sec warmup 16 sec latency 30.307 ms 2024-04-06T03:18:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 8607 18.91 MB/sec warmup 17 sec latency 37.529 ms 2024-04-06T03:18:13.785 INFO:teuthology.orchestra.run.smithi060.stderr:module 'nfs' is already enabled 2024-04-06T03:18:13.819 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:13 smithi060 ceph-mon[24896]: pgmap v244: 97 pgs: 97 active+clean; 140 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 13 MiB/s wr, 863 op/s; 1 B/s, 0 objects/s recovering 2024-04-06T03:18:13.819 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:13 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2817085797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-06T03:18:14.019 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:13 smithi195 ceph-mon[27147]: pgmap v244: 97 pgs: 97 active+clean; 140 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 13 MiB/s wr, 863 op/s; 1 B/s, 0 objects/s recovering 2024-04-06T03:18:14.019 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:13 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2817085797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-06T03:18:14.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 9367 18.19 MB/sec warmup 18 sec latency 43.307 ms 2024-04-06T03:18:14.857 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:14 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2817085797' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-06T03:18:14.857 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:14 smithi060 ceph-mon[24896]: mgrmap e21: smithi060.jttwce(active, since 6m), standbys: smithi195.ultkot 2024-04-06T03:18:15.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-06T03:18:15.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:14 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2817085797' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-06T03:18:15.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:14 smithi195 ceph-mon[27147]: mgrmap e21: smithi060.jttwce(active, since 6m), standbys: smithi195.ultkot 2024-04-06T03:18:15.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 9880 18.34 MB/sec warmup 19 sec latency 43.873 ms 2024-04-06T03:18:15.881 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:15 smithi060 ceph-mon[24896]: pgmap v245: 97 pgs: 97 active+clean; 145 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 11 MiB/s wr, 965 op/s 2024-04-06T03:18:15.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:15 smithi195 ceph-mon[27147]: pgmap v245: 97 pgs: 97 active+clean; 145 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 11 MiB/s wr, 965 op/s 2024-04-06T03:18:16.598 INFO:teuthology.orchestra.run.smithi060.stdout: 5 10476 18.56 MB/sec warmup 20 sec latency 50.771 ms 2024-04-06T03:18:16.895 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:16 smithi060 ceph-mon[24896]: pgmap v246: 97 pgs: 97 active+clean; 145 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 9.5 MiB/s wr, 880 op/s 2024-04-06T03:18:17.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:16 smithi195 ceph-mon[27147]: pgmap v246: 97 pgs: 97 active+clean; 145 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 9.5 MiB/s wr, 880 op/s 2024-04-06T03:18:17.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 10997 18.61 MB/sec warmup 21 sec latency 35.600 ms 2024-04-06T03:18:18.005 INFO:teuthology.orchestra.run.smithi060.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:18:18.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 11542 17.95 MB/sec warmup 22 sec latency 28.470 ms 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: pgmap v247: 97 pgs: 97 active+clean; 146 MiB data, 392 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 793 op/s 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.004 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:19 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: pgmap v247: 97 pgs: 97 active+clean; 146 MiB data, 392 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 793 op/s 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:19.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:19.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:19.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:19 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:19.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 12174 18.14 MB/sec warmup 23 sec latency 22.019 ms 2024-04-06T03:18:19.667 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T03:18:19.667 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:18:19.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-06T03:18:20.185 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:20 smithi060 ceph-mon[24896]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:18:20.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:20 smithi195 ceph-mon[27147]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:18:20.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 12895 17.61 MB/sec warmup 24 sec latency 24.514 ms 2024-04-06T03:18:21.206 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:21 smithi060 ceph-mon[24896]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 413 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 869 op/s 2024-04-06T03:18:21.466 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:21 smithi195 ceph-mon[27147]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 413 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 869 op/s 2024-04-06T03:18:21.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 13279 17.40 MB/sec warmup 25 sec latency 31.228 ms 2024-04-06T03:18:22.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 13820 17.63 MB/sec warmup 26 sec latency 36.246 ms 2024-04-06T03:18:23.228 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:23 smithi060 ceph-mon[24896]: pgmap v249: 97 pgs: 97 active+clean; 156 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 747 op/s 2024-04-06T03:18:23.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:23 smithi195 ceph-mon[27147]: pgmap v249: 97 pgs: 97 active+clean; 156 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 747 op/s 2024-04-06T03:18:23.506 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:18:23.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 14300 17.81 MB/sec warmup 27 sec latency 29.244 ms 2024-04-06T03:18:24.241 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:24 smithi060 ceph-mon[24896]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:24.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:24 smithi195 ceph-mon[27147]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:24.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 14555 17.38 MB/sec warmup 28 sec latency 36.596 ms 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (5m) 30s ago 6m 22.7M - ba2b418f427c 5a135361e3ed 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (6m) 30s ago 6m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (6m) 32s ago 6m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (5m) 30s ago 6m 53.8M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (59s) 30s ago 58s 15.6M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (56s) 32s ago 56s 13.3M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (8m) 30s ago 8m 460M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (6m) 32s ago 6m 415M - 17.2.0 e1d6a67b021e c9b1a21c5cf5 2024-04-06T03:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (8m) 30s ago 8m 48.1M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (6m) 32s ago 6m 39.9M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (41s) 30s ago 40s 47.2M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (35s) 32s ago 35s 46.7M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (6m) 30s ago 6m 17.6M - 1dbe0e931976 2c4375f28350 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (5m) 32s ago 5m 17.8M - 1dbe0e931976 a0369313f174 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (5m) 30s ago 5m 43.1M 3217M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (4m) 30s ago 4m 45.4M 3217M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (4m) 30s ago 4m 41.1M 3217M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (4m) 30s ago 4m 40.4M 3217M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (3m) 32s ago 3m 42.3M 3984M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (3m) 32s ago 3m 42.0M 3984M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (2m) 32s ago 2m 38.6M 3984M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (2m) 32s ago 2m 39.3M 3984M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (5m) 30s ago 5m 62.3M - 514e6a882f6e 03d658f74f0b 2024-04-06T03:18:25.173 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:25 smithi060 ceph-mon[24896]: pgmap v250: 97 pgs: 97 active+clean; 164 MiB data, 482 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 822 op/s 2024-04-06T03:18:25.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:25 smithi060 ceph-mon[24896]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:25.174 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:25 smithi060 ceph-mon[24896]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:25.288 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:18:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-06T03:18:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:18:25.290 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:18:25.390 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:25 smithi195 ceph-mon[27147]: pgmap v250: 97 pgs: 97 active+clean; 164 MiB data, 482 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 822 op/s 2024-04-06T03:18:25.390 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:25 smithi195 ceph-mon[27147]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:25.390 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:25 smithi195 ceph-mon[27147]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:25.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 14901 16.85 MB/sec warmup 29 sec latency 42.159 ms 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "", 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 image" 2024-04-06T03:18:25.877 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:18:26.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:26 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3183953792' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:18:26.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:26 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3183953792' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:18:26.562 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:18:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 15400 16.67 MB/sec warmup 30 sec latency 38.973 ms 2024-04-06T03:18:27.261 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:27 smithi060 ceph-mon[24896]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:27.262 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:27 smithi060 ceph-mon[24896]: pgmap v251: 97 pgs: 97 active+clean; 161 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 688 op/s 2024-04-06T03:18:27.262 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:27 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2128955581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:18:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:27 smithi195 ceph-mon[27147]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:27 smithi195 ceph-mon[27147]: pgmap v251: 97 pgs: 97 active+clean; 161 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 688 op/s 2024-04-06T03:18:27.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:27 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2128955581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:18:27.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 16232 16.64 MB/sec warmup 31 sec latency 19.855 ms 2024-04-06T03:18:28.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 16690 16.40 MB/sec warmup 32 sec latency 33.185 ms 2024-04-06T03:18:29.053 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:29 smithi060 ceph-mon[24896]: pgmap v252: 97 pgs: 97 active+clean; 168 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 736 op/s 2024-04-06T03:18:29.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:29 smithi195 ceph-mon[27147]: pgmap v252: 97 pgs: 97 active+clean; 168 MiB data, 531 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 736 op/s 2024-04-06T03:18:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 17269 16.73 MB/sec warmup 33 sec latency 49.863 ms 2024-04-06T03:18:30.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 17937 17.02 MB/sec warmup 34 sec latency 22.139 ms 2024-04-06T03:18:31.074 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:31 smithi060 ceph-mon[24896]: pgmap v253: 97 pgs: 97 active+clean; 166 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.1 MiB/s wr, 776 op/s 2024-04-06T03:18:31.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:31 smithi195 ceph-mon[27147]: pgmap v253: 97 pgs: 97 active+clean; 166 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.1 MiB/s wr, 776 op/s 2024-04-06T03:18:31.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 18398 16.68 MB/sec warmup 35 sec latency 22.351 ms 2024-04-06T03:18:32.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 18833 16.42 MB/sec warmup 36 sec latency 45.964 ms 2024-04-06T03:18:33.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:33 smithi060 ceph-mon[24896]: pgmap v254: 97 pgs: 97 active+clean; 170 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.7 MiB/s wr, 718 op/s 2024-04-06T03:18:33.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:33 smithi195 ceph-mon[27147]: pgmap v254: 97 pgs: 97 active+clean; 170 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.7 MiB/s wr, 718 op/s 2024-04-06T03:18:33.602 INFO:teuthology.orchestra.run.smithi060.stdout: 5 19529 16.47 MB/sec warmup 37 sec latency 27.918 ms 2024-04-06T03:18:34.599 INFO:teuthology.orchestra.run.smithi060.stdout: 5 20338 16.48 MB/sec warmup 38 sec latency 24.679 ms 2024-04-06T03:18:35.113 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:35 smithi060 ceph-mon[24896]: pgmap v255: 97 pgs: 97 active+clean; 180 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 686 op/s 2024-04-06T03:18:35.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:35 smithi195 ceph-mon[27147]: pgmap v255: 97 pgs: 97 active+clean; 180 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 686 op/s 2024-04-06T03:18:35.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 21152 16.90 MB/sec warmup 39 sec latency 21.407 ms 2024-04-06T03:18:36.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 21653 16.94 MB/sec warmup 40 sec latency 21.196 ms 2024-04-06T03:18:37.114 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:36 smithi060 ceph-mon[24896]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.7 MiB/s wr, 768 op/s 2024-04-06T03:18:37.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:36 smithi195 ceph-mon[27147]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.7 MiB/s wr, 768 op/s 2024-04-06T03:18:37.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 22195 16.65 MB/sec warmup 41 sec latency 36.666 ms 2024-04-06T03:18:38.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 22840 16.70 MB/sec warmup 42 sec latency 22.682 ms 2024-04-06T03:18:38.955 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:38 smithi060 ceph-mon[24896]: pgmap v257: 97 pgs: 97 active+clean; 190 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 851 op/s 2024-04-06T03:18:39.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:38 smithi195 ceph-mon[27147]: pgmap v257: 97 pgs: 97 active+clean; 190 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 851 op/s 2024-04-06T03:18:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 23652 16.58 MB/sec warmup 43 sec latency 18.791 ms 2024-04-06T03:18:40.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 24311 16.80 MB/sec warmup 44 sec latency 24.040 ms 2024-04-06T03:18:41.137 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:40 smithi060 ceph-mon[24896]: pgmap v258: 97 pgs: 97 active+clean; 194 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.3 MiB/s wr, 888 op/s 2024-04-06T03:18:41.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:40 smithi195 ceph-mon[27147]: pgmap v258: 97 pgs: 97 active+clean; 194 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.3 MiB/s wr, 888 op/s 2024-04-06T03:18:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 25012 17.03 MB/sec warmup 45 sec latency 24.057 ms 2024-04-06T03:18:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 25379 16.84 MB/sec warmup 46 sec latency 33.099 ms 2024-04-06T03:18:43.150 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:42 smithi060 ceph-mon[24896]: pgmap v259: 97 pgs: 97 active+clean; 194 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 9.5 MiB/s wr, 891 op/s 2024-04-06T03:18:43.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:42 smithi195 ceph-mon[27147]: pgmap v259: 97 pgs: 97 active+clean; 194 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 9.5 MiB/s wr, 891 op/s 2024-04-06T03:18:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 25844 16.59 MB/sec warmup 47 sec latency 33.251 ms 2024-04-06T03:18:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 26461 16.61 MB/sec warmup 48 sec latency 56.036 ms 2024-04-06T03:18:45.184 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:44 smithi060 ceph-mon[24896]: pgmap v260: 97 pgs: 97 active+clean; 202 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.6 MiB/s wr, 947 op/s 2024-04-06T03:18:45.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:44 smithi195 ceph-mon[27147]: pgmap v260: 97 pgs: 97 active+clean; 202 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.6 MiB/s wr, 947 op/s 2024-04-06T03:18:45.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 27137 16.46 MB/sec warmup 49 sec latency 39.655 ms 2024-04-06T03:18:46.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 27841 16.67 MB/sec warmup 50 sec latency 23.483 ms 2024-04-06T03:18:46.958 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:46 smithi060 ceph-mon[24896]: pgmap v261: 97 pgs: 97 active+clean; 203 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 816 op/s 2024-04-06T03:18:47.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:46 smithi195 ceph-mon[27147]: pgmap v261: 97 pgs: 97 active+clean; 203 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 816 op/s 2024-04-06T03:18:47.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 28596 16.93 MB/sec warmup 51 sec latency 22.933 ms 2024-04-06T03:18:48.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 29173 16.79 MB/sec warmup 52 sec latency 40.254 ms 2024-04-06T03:18:48.968 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:48 smithi060 ceph-mon[24896]: pgmap v262: 97 pgs: 97 active+clean; 207 MiB data, 842 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 677 op/s 2024-04-06T03:18:49.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:48 smithi195 ceph-mon[27147]: pgmap v262: 97 pgs: 97 active+clean; 207 MiB data, 842 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 677 op/s 2024-04-06T03:18:49.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 29748 16.70 MB/sec warmup 53 sec latency 34.008 ms 2024-04-06T03:18:50.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 30470 16.66 MB/sec warmup 54 sec latency 47.584 ms 2024-04-06T03:18:51.020 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:50 smithi060 ceph-mon[24896]: pgmap v263: 97 pgs: 97 active+clean; 214 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 840 op/s 2024-04-06T03:18:51.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:50 smithi195 ceph-mon[27147]: pgmap v263: 97 pgs: 97 active+clean; 214 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 840 op/s 2024-04-06T03:18:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 31257 16.83 MB/sec warmup 55 sec latency 48.679 ms 2024-04-06T03:18:52.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 32165 17.14 MB/sec warmup 56 sec latency 15.196 ms 2024-04-06T03:18:53.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 32934 17.09 MB/sec warmup 57 sec latency 19.608 ms 2024-04-06T03:18:53.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: pgmap v264: 97 pgs: 97 active+clean; 214 MiB data, 887 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 769 op/s 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:53.987 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:53 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi060.jttwce) 2024-04-06T03:18:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: pgmap v264: 97 pgs: 97 active+clean; 214 MiB data, 887 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 769 op/s 2024-04-06T03:18:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T03:18:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T03:18:54.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:54.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:18:54.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:53 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi060.jttwce) 2024-04-06T03:18:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: 5 33974 17.16 MB/sec warmup 58 sec latency 20.120 ms 2024-04-06T03:18:54.992 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:54 smithi060 ceph-mon[24896]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi195 2024-04-06T03:18:55.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:54 smithi195 ceph-mon[27147]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi195 2024-04-06T03:18:55.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 34770 17.26 MB/sec warmup 59 sec latency 40.511 ms 2024-04-06T03:18:55.967 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:55 smithi195 ceph-mon[27147]: pgmap v265: 97 pgs: 97 active+clean; 220 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 824 op/s 2024-04-06T03:18:56.008 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:55 smithi060 ceph-mon[24896]: pgmap v265: 97 pgs: 97 active+clean; 220 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 824 op/s 2024-04-06T03:18:56.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 35546 17.52 MB/sec warmup 60 sec latency 26.128 ms 2024-04-06T03:18:57.017 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:56 smithi060 ceph-mon[24896]: pgmap v266: 97 pgs: 97 active+clean; 232 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 848 op/s 2024-04-06T03:18:57.061 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:18:57.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:56 smithi195 ceph-mon[27147]: pgmap v266: 97 pgs: 97 active+clean; 232 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 848 op/s 2024-04-06T03:18:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 36109 17.42 MB/sec warmup 61 sec latency 38.903 ms 2024-04-06T03:18:58.022 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:57 smithi060 ceph-mon[24896]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:58.022 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:57 smithi060 ceph-mon[24896]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:58.035 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:18:58.035 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (5m) 63s ago 7m 22.7M - ba2b418f427c 5a135361e3ed 2024-04-06T03:18:58.035 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (7m) 63s ago 7m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (6m) 66s ago 6m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (5m) 63s ago 7m 53.8M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (93s) 63s ago 91s 15.6M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (89s) 66s ago 89s 13.3M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (8m) 63s ago 8m 460M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (6m) 66s ago 6m 415M - 17.2.0 e1d6a67b021e c9b1a21c5cf5 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (8m) 63s ago 9m 48.1M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (6m) 66s ago 6m 39.9M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (74s) 63s ago 74s 47.2M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (69s) 66s ago 68s 46.7M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (7m) 63s ago 7m 17.6M - 1dbe0e931976 2c4375f28350 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (6m) 66s ago 6m 17.8M - 1dbe0e931976 a0369313f174 2024-04-06T03:18:58.036 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (6m) 63s ago 6m 43.1M 3217M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (5m) 63s ago 5m 45.4M 3217M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (5m) 63s ago 5m 41.1M 3217M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (4m) 63s ago 4m 40.4M 3217M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (4m) 66s ago 4m 42.3M 3984M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (3m) 66s ago 3m 42.0M 3984M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (3m) 66s ago 3m 38.6M 3984M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (3m) 66s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:18:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (6m) 63s ago 6m 62.3M - 514e6a882f6e 03d658f74f0b 2024-04-06T03:18:58.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:57 smithi195 ceph-mon[27147]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:58.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:57 smithi195 ceph-mon[27147]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 36727 17.31 MB/sec warmup 62 sec latency 24.751 ms 2024-04-06T03:18:58.674 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:18:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:18:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:18:58.675 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:18:59.025 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:58 smithi060 ceph-mon[24896]: pgmap v267: 97 pgs: 97 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 874 op/s 2024-04-06T03:18:59.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:18:59.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:18:59.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:58 smithi060 ceph-mon[24896]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:59.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:58 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2408499274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:18:59.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:58 smithi195 ceph-mon[27147]: pgmap v267: 97 pgs: 97 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.8 MiB/s wr, 874 op/s 2024-04-06T03:18:59.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:18:59.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:18:59.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:58 smithi195 ceph-mon[27147]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:59.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:58 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2408499274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:18:59.227 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:18:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:18:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:18:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:18:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "0/23 daemons upgraded", 2024-04-06T03:18:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c image on host smithi195" 2024-04-06T03:18:59.228 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:18:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 37444 17.31 MB/sec warmup 63 sec latency 19.776 ms 2024-04-06T03:18:59.882 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:19:00.029 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:59 smithi060 ceph-mon[24896]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:00.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:18:59 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3008416707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:19:00.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:59 smithi195 ceph-mon[27147]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:00.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:18:59 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3008416707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:19:00.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 38107 17.25 MB/sec warmup 64 sec latency 39.596 ms 2024-04-06T03:19:01.037 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:00 smithi060 ceph-mon[24896]: pgmap v268: 97 pgs: 97 active+clean; 241 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-04-06T03:19:01.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:00 smithi195 ceph-mon[27147]: pgmap v268: 97 pgs: 97 active+clean; 241 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-04-06T03:19:01.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 38922 17.51 MB/sec warmup 65 sec latency 17.922 ms 2024-04-06T03:19:02.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 39529 17.52 MB/sec warmup 66 sec latency 19.554 ms 2024-04-06T03:19:03.053 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:02 smithi060 ceph-mon[24896]: pgmap v269: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-04-06T03:19:03.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:02 smithi195 ceph-mon[27147]: pgmap v269: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-04-06T03:19:03.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 40124 17.40 MB/sec warmup 67 sec latency 22.186 ms 2024-04-06T03:19:04.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 40825 17.38 MB/sec warmup 68 sec latency 17.880 ms 2024-04-06T03:19:05.070 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:04 smithi060 ceph-mon[24896]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-04-06T03:19:05.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:04 smithi195 ceph-mon[27147]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-04-06T03:19:05.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 41629 17.38 MB/sec warmup 69 sec latency 20.909 ms 2024-04-06T03:19:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 42362 17.57 MB/sec warmup 70 sec latency 20.863 ms 2024-04-06T03:19:07.094 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:06 smithi060 ceph-mon[24896]: pgmap v271: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 990 op/s 2024-04-06T03:19:07.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:06 smithi195 ceph-mon[27147]: pgmap v271: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 990 op/s 2024-04-06T03:19:07.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 42970 17.62 MB/sec warmup 71 sec latency 32.287 ms 2024-04-06T03:19:08.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 43412 17.48 MB/sec warmup 72 sec latency 35.077 ms 2024-04-06T03:19:08.919 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:08 smithi060 ceph-mon[24896]: pgmap v272: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 884 op/s 2024-04-06T03:19:09.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:08 smithi195 ceph-mon[27147]: pgmap v272: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 9.8 MiB/s wr, 884 op/s 2024-04-06T03:19:09.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 43848 17.30 MB/sec warmup 73 sec latency 33.172 ms 2024-04-06T03:19:10.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 44363 17.27 MB/sec warmup 74 sec latency 28.647 ms 2024-04-06T03:19:11.131 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:10 smithi060 ceph-mon[24896]: pgmap v273: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-06T03:19:11.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:10 smithi195 ceph-mon[27147]: pgmap v273: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-06T03:19:11.601 INFO:teuthology.orchestra.run.smithi060.stdout: 5 45027 17.22 MB/sec warmup 75 sec latency 31.278 ms 2024-04-06T03:19:12.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 45939 17.45 MB/sec warmup 76 sec latency 18.382 ms 2024-04-06T03:19:13.144 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:12 smithi060 ceph-mon[24896]: pgmap v274: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 859 op/s 2024-04-06T03:19:13.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:12 smithi195 ceph-mon[27147]: pgmap v274: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 859 op/s 2024-04-06T03:19:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 46735 17.53 MB/sec warmup 77 sec latency 21.682 ms 2024-04-06T03:19:14.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 47421 17.45 MB/sec warmup 78 sec latency 24.315 ms 2024-04-06T03:19:15.165 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:14 smithi060 ceph-mon[24896]: pgmap v275: 97 pgs: 97 active+clean; 263 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 915 op/s 2024-04-06T03:19:15.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:14 smithi195 ceph-mon[27147]: pgmap v275: 97 pgs: 97 active+clean; 263 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 915 op/s 2024-04-06T03:19:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 47999 17.40 MB/sec warmup 79 sec latency 46.377 ms 2024-04-06T03:19:16.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 48965 17.50 MB/sec warmup 80 sec latency 29.052 ms 2024-04-06T03:19:16.921 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:16 smithi060 ceph-mon[24896]: pgmap v276: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 782 op/s 2024-04-06T03:19:17.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:16 smithi195 ceph-mon[27147]: pgmap v276: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 782 op/s 2024-04-06T03:19:17.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 49878 17.72 MB/sec warmup 81 sec latency 22.843 ms 2024-04-06T03:19:18.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 50567 17.69 MB/sec warmup 82 sec latency 18.986 ms 2024-04-06T03:19:19.183 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:18 smithi060 ceph-mon[24896]: pgmap v277: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 825 op/s 2024-04-06T03:19:19.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:18 smithi195 ceph-mon[27147]: pgmap v277: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 825 op/s 2024-04-06T03:19:19.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 51400 17.69 MB/sec warmup 83 sec latency 13.937 ms 2024-04-06T03:19:20.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 52274 17.73 MB/sec warmup 84 sec latency 26.631 ms 2024-04-06T03:19:21.195 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:20 smithi060 ceph-mon[24896]: pgmap v278: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 989 op/s 2024-04-06T03:19:21.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:20 smithi195 ceph-mon[27147]: pgmap v278: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 989 op/s 2024-04-06T03:19:21.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 52982 17.82 MB/sec warmup 85 sec latency 20.552 ms 2024-04-06T03:19:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 53600 17.84 MB/sec warmup 86 sec latency 39.459 ms 2024-04-06T03:19:22.961 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:22 smithi060 ceph-mon[24896]: pgmap v279: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-06T03:19:23.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:22 smithi195 ceph-mon[27147]: pgmap v279: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-06T03:19:23.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 54119 17.77 MB/sec warmup 87 sec latency 22.039 ms 2024-04-06T03:19:24.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 54714 17.68 MB/sec warmup 88 sec latency 21.459 ms 2024-04-06T03:19:24.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:24 smithi060 ceph-mon[24896]: pgmap v280: 97 pgs: 97 active+clean; 296 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-04-06T03:19:25.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:24 smithi195 ceph-mon[27147]: pgmap v280: 97 pgs: 97 active+clean; 296 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-04-06T03:19:25.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 55346 17.70 MB/sec warmup 89 sec latency 36.261 ms 2024-04-06T03:19:26.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 56051 17.68 MB/sec warmup 90 sec latency 42.286 ms 2024-04-06T03:19:26.922 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:26 smithi060 ceph-mon[24896]: pgmap v281: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-06T03:19:27.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:26 smithi195 ceph-mon[27147]: pgmap v281: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-06T03:19:27.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 56602 17.72 MB/sec warmup 91 sec latency 29.660 ms 2024-04-06T03:19:28.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 57173 17.73 MB/sec warmup 92 sec latency 33.585 ms 2024-04-06T03:19:29.192 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:28 smithi060 ceph-mon[24896]: pgmap v282: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 935 op/s 2024-04-06T03:19:29.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:28 smithi195 ceph-mon[27147]: pgmap v282: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 935 op/s 2024-04-06T03:19:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: 5 57573 17.65 MB/sec warmup 93 sec latency 38.341 ms 2024-04-06T03:19:30.361 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:19:30.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 58064 17.54 MB/sec warmup 94 sec latency 36.254 ms 2024-04-06T03:19:31.212 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:30 smithi060 ceph-mon[24896]: pgmap v283: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-06T03:19:31.212 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:30 smithi060 ceph-mon[24896]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:31.212 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:30 smithi060 ceph-mon[24896]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:31.212 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:19:31.212 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (6m) 97s ago 8m 22.7M - ba2b418f427c 5a135361e3ed 2024-04-06T03:19:31.212 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (7m) 97s ago 7m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (7m) 99s ago 7m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (6m) 97s ago 7m 53.8M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (2m) 97s ago 2m 15.6M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (2m) 99s ago 2m 13.3M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (9m) 97s ago 9m 460M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (7m) 99s ago 7m 415M - 17.2.0 e1d6a67b021e c9b1a21c5cf5 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (9m) 97s ago 9m 48.1M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (7m) 99s ago 7m 39.9M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (107s) 97s ago 107s 47.2M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (102s) 99s ago 101s 46.7M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (7m) 97s ago 7m 17.6M - 1dbe0e931976 2c4375f28350 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (7m) 99s ago 7m 17.8M - 1dbe0e931976 a0369313f174 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (6m) 97s ago 6m 43.1M 3217M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (6m) 97s ago 6m 45.4M 3217M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:19:31.213 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (5m) 97s ago 5m 41.1M 3217M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (5m) 97s ago 5m 40.4M 3217M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (4m) 99s ago 4m 42.3M 3984M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (4m) 99s ago 4m 42.0M 3984M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (3m) 99s ago 3m 38.6M 3984M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (3m) 99s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:19:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (6m) 97s ago 6m 62.3M - 514e6a882f6e 03d658f74f0b 2024-04-06T03:19:31.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:30 smithi195 ceph-mon[27147]: pgmap v283: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-06T03:19:31.217 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:30 smithi195 ceph-mon[27147]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:31.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:30 smithi195 ceph-mon[27147]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:31.603 INFO:teuthology.orchestra.run.smithi060.stdout: 5 58451 17.43 MB/sec warmup 95 sec latency 50.849 ms 2024-04-06T03:19:31.696 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:19:31.696 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:19:31.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:19:31.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:19:31.697 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:19:31.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:31 smithi060 ceph-mon[24896]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:31.980 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:31 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1133475576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "0/23 daemons upgraded", 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c image on host smithi195" 2024-04-06T03:19:32.192 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:19:32.467 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:31 smithi195 ceph-mon[27147]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:32.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:31 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1133475576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:19:32.604 INFO:teuthology.orchestra.run.smithi060.stdout: 5 59091 17.43 MB/sec warmup 96 sec latency 36.739 ms 2024-04-06T03:19:32.793 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:19:32.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:32 smithi060 ceph-mon[24896]: pgmap v284: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 915 op/s 2024-04-06T03:19:32.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:32 smithi060 ceph-mon[24896]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:32.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:32 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/4195967652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:19:33.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:32 smithi195 ceph-mon[27147]: pgmap v284: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 915 op/s 2024-04-06T03:19:33.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:32 smithi195 ceph-mon[27147]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:33.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:32 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/4195967652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:19:33.604 INFO:teuthology.orchestra.run.smithi060.stdout: 5 59896 17.54 MB/sec warmup 97 sec latency 23.886 ms 2024-04-06T03:19:34.604 INFO:teuthology.orchestra.run.smithi060.stdout: 5 60595 17.58 MB/sec warmup 98 sec latency 17.691 ms 2024-04-06T03:19:34.995 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:34 smithi060 ceph-mon[24896]: pgmap v285: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 952 op/s 2024-04-06T03:19:35.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:34 smithi195 ceph-mon[27147]: pgmap v285: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 952 op/s 2024-04-06T03:19:35.604 INFO:teuthology.orchestra.run.smithi060.stdout: 5 61333 17.56 MB/sec warmup 99 sec latency 20.748 ms 2024-04-06T03:19:36.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 62104 17.55 MB/sec warmup 100 sec latency 28.391 ms 2024-04-06T03:19:37.006 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:36 smithi060 ceph-mon[24896]: pgmap v286: 97 pgs: 97 active+clean; 315 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 769 op/s 2024-04-06T03:19:37.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:36 smithi195 ceph-mon[27147]: pgmap v286: 97 pgs: 97 active+clean; 315 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 769 op/s 2024-04-06T03:19:37.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 62827 17.58 MB/sec warmup 101 sec latency 26.336 ms 2024-04-06T03:19:38.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 63551 17.65 MB/sec warmup 102 sec latency 25.016 ms 2024-04-06T03:19:39.025 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:38 smithi060 ceph-mon[24896]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 732 op/s 2024-04-06T03:19:39.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:38 smithi195 ceph-mon[27147]: pgmap v287: 97 pgs: 97 active+clean; 317 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 732 op/s 2024-04-06T03:19:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 64131 17.66 MB/sec warmup 103 sec latency 19.152 ms 2024-04-06T03:19:40.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 64693 17.62 MB/sec warmup 104 sec latency 29.517 ms 2024-04-06T03:19:41.046 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:40 smithi060 ceph-mon[24896]: pgmap v288: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.6 MiB/s wr, 905 op/s 2024-04-06T03:19:41.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:40 smithi195 ceph-mon[27147]: pgmap v288: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 9.6 MiB/s wr, 905 op/s 2024-04-06T03:19:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 65144 17.52 MB/sec warmup 105 sec latency 122.285 ms 2024-04-06T03:19:42.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 65826 17.51 MB/sec warmup 106 sec latency 32.772 ms 2024-04-06T03:19:43.141 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:42 smithi060 ceph-mon[24896]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.3 MiB/s wr, 909 op/s 2024-04-06T03:19:43.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:42 smithi195 ceph-mon[27147]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.3 MiB/s wr, 909 op/s 2024-04-06T03:19:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 66359 17.48 MB/sec warmup 107 sec latency 66.874 ms 2024-04-06T03:19:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 66929 17.49 MB/sec warmup 108 sec latency 32.063 ms 2024-04-06T03:19:45.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:44 smithi060 ceph-mon[24896]: pgmap v290: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 938 op/s 2024-04-06T03:19:45.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:44 smithi195 ceph-mon[27147]: pgmap v290: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 938 op/s 2024-04-06T03:19:45.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 67406 17.47 MB/sec warmup 109 sec latency 78.932 ms 2024-04-06T03:19:46.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 67792 17.43 MB/sec warmup 110 sec latency 190.707 ms 2024-04-06T03:19:46.936 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:46 smithi060 ceph-mon[24896]: pgmap v291: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 849 op/s 2024-04-06T03:19:47.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:46 smithi195 ceph-mon[27147]: pgmap v291: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 849 op/s 2024-04-06T03:19:47.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 68210 17.36 MB/sec warmup 111 sec latency 65.762 ms 2024-04-06T03:19:48.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 68615 17.28 MB/sec warmup 112 sec latency 75.506 ms 2024-04-06T03:19:49.009 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:48 smithi060 ceph-mon[24896]: pgmap v292: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 776 op/s 2024-04-06T03:19:49.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:48 smithi195 ceph-mon[27147]: pgmap v292: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 776 op/s 2024-04-06T03:19:49.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 69051 17.19 MB/sec warmup 113 sec latency 50.525 ms 2024-04-06T03:19:50.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 69538 17.14 MB/sec warmup 114 sec latency 41.163 ms 2024-04-06T03:19:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 70053 17.15 MB/sec warmup 115 sec latency 29.513 ms 2024-04-06T03:19:51.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: pgmap v293: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 946 op/s 2024-04-06T03:19:51.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: Upgrade: Updating mgr.smithi195.ultkot 2024-04-06T03:19:51.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:51.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:19:51.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:19:51.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:51.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:51 smithi060 ceph-mon[24896]: Deploying daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: pgmap v293: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 946 op/s 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: Upgrade: Updating mgr.smithi195.ultkot 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:19:52.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:52.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:51 smithi195 ceph-mon[27147]: Deploying daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:19:52.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 70660 17.17 MB/sec warmup 116 sec latency 32.440 ms 2024-04-06T03:19:53.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 71161 17.17 MB/sec warmup 117 sec latency 26.745 ms 2024-04-06T03:19:53.837 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:53 smithi060 ceph-mon[24896]: pgmap v294: 97 pgs: 97 active+clean; 325 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.5 MiB/s wr, 776 op/s 2024-04-06T03:19:53.968 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:53 smithi195 ceph-mon[27147]: pgmap v294: 97 pgs: 97 active+clean; 325 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.5 MiB/s wr, 776 op/s 2024-04-06T03:19:54.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 71801 17.15 MB/sec warmup 118 sec latency 22.292 ms 2024-04-06T03:19:55.599 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:55 smithi060 ceph-mon[24896]: pgmap v295: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 767 op/s 2024-04-06T03:19:55.599 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:55.599 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:55.599 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:55.599 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:55 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:55.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 72304 17.08 MB/sec warmup 119 sec latency 29.882 ms 2024-04-06T03:19:55.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:55 smithi195 ceph-mon[27147]: pgmap v295: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 767 op/s 2024-04-06T03:19:55.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:55.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:55.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:55.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:55 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:56.968 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:56 smithi195 ceph-mon[27147]: pgmap v296: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-04-06T03:19:57.126 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:56 smithi060 ceph-mon[24896]: pgmap v296: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-04-06T03:19:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 73349 14.41 MB/sec execute 1 sec latency 51.125 ms 2024-04-06T03:19:58.046 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:57 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:19:58.136 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:57 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:19:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 73810 13.94 MB/sec execute 2 sec latency 31.024 ms 2024-04-06T03:19:59.095 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:58 smithi195 ceph-mon[27147]: pgmap v297: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-04-06T03:19:59.095 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:19:59.095 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:19:58 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:59.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:58 smithi060 ceph-mon[24896]: pgmap v297: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-04-06T03:19:59.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:19:59.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:19:58 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:19:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 74238 14.77 MB/sec execute 3 sec latency 37.478 ms 2024-04-06T03:20:00.412 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:00 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:00.412 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:00 smithi060 ceph-mon[24896]: overall HEALTH_OK 2024-04-06T03:20:00.413 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:00 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:00.414 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:00 smithi195 ceph-mon[27147]: overall HEALTH_OK 2024-04-06T03:20:00.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 74688 15.00 MB/sec execute 4 sec latency 36.248 ms 2024-04-06T03:20:01.297 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:01 smithi195 ceph-mon[27147]: pgmap v298: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-04-06T03:20:01.416 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:01 smithi060 ceph-mon[24896]: pgmap v298: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-04-06T03:20:01.605 INFO:teuthology.orchestra.run.smithi060.stdout: 5 75271 14.92 MB/sec execute 5 sec latency 38.019 ms 2024-04-06T03:20:02.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 75786 13.99 MB/sec execute 6 sec latency 24.727 ms 2024-04-06T03:20:03.320 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:20:03.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: pgmap v299: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.5 MiB/s wr, 723 op/s 2024-04-06T03:20:03.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:03.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr fail", "who": "smithi060.jttwce"}]: dispatch 2024-04-06T03:20:03.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:03 smithi060 ceph-mon[24896]: osdmap e70: 8 total, 8 up, 8 in 2024-04-06T03:20:03.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 76190 13.41 MB/sec execute 7 sec latency 33.562 ms 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: pgmap v299: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.5 MiB/s wr, 723 op/s 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr fail", "who": "smithi060.jttwce"}]: dispatch 2024-04-06T03:20:03.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:03 smithi195 ceph-mon[27147]: osdmap e70: 8 total, 8 up, 8 in 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi060.jttwce) 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: Failing over to other MGR 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: Active manager daemon smithi195.ultkot restarted 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: Activating manager daemon smithi195.ultkot 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: osdmap e71: 8 total, 8 up, 8 in 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:20:04.265 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "mgr fail", "who": "smithi060.jttwce"}]': finished 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: mgrmap e22: smithi195.ultkot(active, starting, since 0.85027s) 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:20:04.266 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:20:04.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:04.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:20:04.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:20:04.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:20:04.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:04 smithi060 ceph-mon[24896]: Manager daemon smithi195.ultkot is now available 2024-04-06T03:20:04.417 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi060.jttwce) 2024-04-06T03:20:04.417 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: Failing over to other MGR 2024-04-06T03:20:04.417 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:04.417 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: Active manager daemon smithi195.ultkot restarted 2024-04-06T03:20:04.417 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: Activating manager daemon smithi195.ultkot 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: osdmap e71: 8 total, 8 up, 8 in 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.14204 172.21.15.60:0/2815517012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "mgr fail", "who": "smithi060.jttwce"}]': finished 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: mgrmap e22: smithi195.ultkot(active, starting, since 0.85027s) 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:20:04.418 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:20:04.419 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:04 smithi195 ceph-mon[27147]: Manager daemon smithi195.ultkot is now available 2024-04-06T03:20:04.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 76602 12.71 MB/sec execute 8 sec latency 37.737 ms 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (7m) 2m ago 8m 22.7M - ba2b418f427c 5a135361e3ed 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (8m) 2m ago 8m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (7m) 6s ago 7m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (6m) 2m ago 8m 53.8M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (2m) 2m ago 2m 15.6M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (2m) 6s ago 2m 13.8M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (11s) 6s ago 7m 47.9M - 19.0.0-2027-g9f192350 a5511c7deba9 3ab7c8547fa3 2024-04-06T03:20:05.208 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (9m) 2m ago 10m 48.1M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (7m) 6s ago 7m 40.9M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (2m) 2m ago 2m 47.2M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (2m) 6s ago 2m 73.0M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (8m) 2m ago 8m 17.6M - 1dbe0e931976 2c4375f28350 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (7m) 6s ago 7m 17.4M - 1dbe0e931976 a0369313f174 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (7m) 2m ago 7m 43.1M 3217M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (6m) 2m ago 6m 45.4M 3217M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (6m) 2m ago 6m 41.1M 3217M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (5m) 2m ago 5m 40.4M 3217M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (5m) 6s ago 5m 260M 3984M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (4m) 6s ago 4m 295M 3984M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (4m) 6s ago 4m 265M 3984M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (4m) 6s ago 4m 373M 3984M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:20:05.209 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (7m) 2m ago 7m 62.3M - 514e6a882f6e 03d658f74f0b 2024-04-06T03:20:05.458 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.458 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:05.458 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/trash_purge_schedule"}]: dispatch 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/trash_purge_schedule"}]: dispatch 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:05 smithi060 ceph-mon[24896]: mgrmap e23: smithi195.ultkot(active, since 1.85234s) 2024-04-06T03:20:05.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 77008 12.47 MB/sec execute 9 sec latency 44.954 ms 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/trash_purge_schedule"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi195.ultkot/trash_purge_schedule"}]: dispatch 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:05.647 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:05 smithi195 ceph-mon[27147]: mgrmap e23: smithi195.ultkot(active, since 1.85234s) 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:20:05.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:05.777 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:20:05.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-06T03:20:05.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:20:05.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:20:05.777 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:20:06.299 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:20:06.300 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:20:06.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:06 smithi060 ceph-mon[24896]: Deploying cephadm binary to smithi195 2024-04-06T03:20:06.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:06 smithi060 ceph-mon[24896]: Deploying cephadm binary to smithi060 2024-04-06T03:20:06.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:06 smithi060 ceph-mon[24896]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:06.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:06 smithi060 ceph-mon[24896]: pgmap v3: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:06.495 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:06 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1169491688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:20:06.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 77450 13.18 MB/sec execute 10 sec latency 39.968 ms 2024-04-06T03:20:06.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:06 smithi195 ceph-mon[27147]: Deploying cephadm binary to smithi195 2024-04-06T03:20:06.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:06 smithi195 ceph-mon[27147]: Deploying cephadm binary to smithi060 2024-04-06T03:20:06.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:06 smithi195 ceph-mon[27147]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:06.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:06 smithi195 ceph-mon[27147]: pgmap v3: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:06.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:06 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1169491688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:20:07.040 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:20:05] ENGINE Bus STARTING 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:20:05] ENGINE Serving on http://172.21.15.195:8765 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:20:06] ENGINE Serving on https://172.21.15.195:7150 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:20:06] ENGINE Bus STARTED 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: pgmap v4: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: mgrmap e24: smithi195.ultkot(active, since 2s) 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:07.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:07 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2323846400' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:20:07.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 77943 13.35 MB/sec execute 11 sec latency 38.771 ms 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: [06/Apr/2024:03:20:05] ENGINE Bus STARTING 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: [06/Apr/2024:03:20:05] ENGINE Serving on http://172.21.15.195:8765 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: [06/Apr/2024:03:20:06] ENGINE Serving on https://172.21.15.195:7150 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: [06/Apr/2024:03:20:06] ENGINE Bus STARTED 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: pgmap v4: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:07.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: mgrmap e24: smithi195.ultkot(active, since 2s) 2024-04-06T03:20:07.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:07.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:07 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2323846400' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:20:08.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 78217 12.82 MB/sec execute 12 sec latency 83.589 ms 2024-04-06T03:20:09.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:09 smithi060 ceph-mon[24896]: pgmap v5: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:09.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:09 smithi060 ceph-mon[24896]: mgrmap e25: smithi195.ultkot(active, since 4s) 2024-04-06T03:20:09.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 78455 12.35 MB/sec execute 13 sec latency 85.830 ms 2024-04-06T03:20:09.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:09 smithi195 ceph-mon[27147]: pgmap v5: 97 pgs: 97 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:20:09.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:09 smithi195 ceph-mon[27147]: mgrmap e25: smithi195.ultkot(active, since 4s) 2024-04-06T03:20:10.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 79001 12.33 MB/sec execute 14 sec latency 37.824 ms 2024-04-06T03:20:11.515 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:11 smithi060 ceph-mon[24896]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s wr, 224 op/s 2024-04-06T03:20:11.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 79411 12.10 MB/sec execute 15 sec latency 30.931 ms 2024-04-06T03:20:11.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:11 smithi195 ceph-mon[27147]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s wr, 224 op/s 2024-04-06T03:20:12.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 79853 11.95 MB/sec execute 16 sec latency 43.515 ms 2024-04-06T03:20:13.544 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: pgmap v7: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 4.6 MiB/s wr, 421 op/s 2024-04-06T03:20:13.544 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: Standby manager daemon smithi060.jttwce started 2024-04-06T03:20:13.544 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/crt"}]: dispatch 2024-04-06T03:20:13.544 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:20:13.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/key"}]: dispatch 2024-04-06T03:20:13.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:13 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:20:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 80392 11.89 MB/sec execute 17 sec latency 31.995 ms 2024-04-06T03:20:13.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: pgmap v7: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 4.6 MiB/s wr, 421 op/s 2024-04-06T03:20:13.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: Standby manager daemon smithi060.jttwce started 2024-04-06T03:20:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/crt"}]: dispatch 2024-04-06T03:20:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:20:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/key"}]: dispatch 2024-04-06T03:20:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:13 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/2767118380' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:20:14.555 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:14 smithi060 ceph-mon[24896]: mgrmap e26: smithi195.ultkot(active, since 10s), standbys: smithi060.jttwce 2024-04-06T03:20:14.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:14 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:20:14.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 80863 12.28 MB/sec execute 18 sec latency 39.313 ms 2024-04-06T03:20:14.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:14 smithi195 ceph-mon[27147]: mgrmap e26: smithi195.ultkot(active, since 10s), standbys: smithi060.jttwce 2024-04-06T03:20:14.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:14 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:20:15.593 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:15 smithi060 ceph-mon[24896]: pgmap v8: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-04-06T03:20:15.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 81348 12.40 MB/sec execute 19 sec latency 81.200 ms 2024-04-06T03:20:15.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:15 smithi195 ceph-mon[27147]: pgmap v8: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-04-06T03:20:16.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 81677 12.35 MB/sec execute 20 sec latency 76.774 ms 2024-04-06T03:20:17.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:16 smithi060 ceph-mon[24896]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 569 op/s 2024-04-06T03:20:17.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:16 smithi195 ceph-mon[27147]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 569 op/s 2024-04-06T03:20:17.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 81957 12.14 MB/sec execute 21 sec latency 55.601 ms 2024-04-06T03:20:18.140 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:18 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:18.141 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:18 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:18.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:18 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:18.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:18 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:18.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 82327 11.95 MB/sec execute 22 sec latency 38.831 ms 2024-04-06T03:20:19.147 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:19 smithi060 ceph-mon[24896]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.3 MiB/s wr, 519 op/s 2024-04-06T03:20:19.148 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:19 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.148 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:19 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.148 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:19 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.148 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:19 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.422 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:19 smithi195 ceph-mon[27147]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.3 MiB/s wr, 519 op/s 2024-04-06T03:20:19.422 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:19 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.422 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:19 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.422 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:19 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.422 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:19 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:19.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 82706 11.75 MB/sec execute 23 sec latency 64.968 ms 2024-04-06T03:20:20.171 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:20 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:20.406 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:20 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:20.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 83068 11.59 MB/sec execute 24 sec latency 31.331 ms 2024-04-06T03:20:21.200 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: pgmap v11: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 535 op/s 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.201 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:21 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: pgmap v11: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.6 MiB/s wr, 535 op/s 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:21.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:21 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:21.606 INFO:teuthology.orchestra.run.smithi060.stdout: 5 83646 11.69 MB/sec execute 25 sec latency 35.449 ms 2024-04-06T03:20:22.229 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:22 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi060 to 1937M 2024-04-06T03:20:22.229 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:22 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:22.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:22 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi060 to 1937M 2024-04-06T03:20:22.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:22 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: 5 84153 11.81 MB/sec execute 26 sec latency 70.537 ms 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: Adjusting osd_memory_target on smithi195 to 2704M 2024-04-06T03:20:23.235 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:23.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:23.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:23.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:20:23.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:20:23.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:23 smithi060 ceph-mon[24896]: pgmap v12: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.8 MiB/s wr, 655 op/s 2024-04-06T03:20:23.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:23.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: Adjusting osd_memory_target on smithi195 to 2704M 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:20:23.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:23 smithi195 ceph-mon[27147]: pgmap v12: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.8 MiB/s wr, 655 op/s 2024-04-06T03:20:23.607 INFO:teuthology.orchestra.run.smithi060.stdout: 5 84647 12.04 MB/sec execute 27 sec latency 21.633 ms 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:24 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:24 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:24.607 INFO:teuthology.orchestra.run.smithi060.stdout: 5 85204 12.22 MB/sec execute 28 sec latency 42.650 ms 2024-04-06T03:20:25.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:25 smithi060 ceph-mon[24896]: Reconfiguring alertmanager.smithi060 (dependencies changed)... 2024-04-06T03:20:25.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:25 smithi060 ceph-mon[24896]: Reconfiguring daemon alertmanager.smithi060 on smithi060 2024-04-06T03:20:25.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:25 smithi060 ceph-mon[24896]: pgmap v13: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 503 op/s 2024-04-06T03:20:25.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:25 smithi195 ceph-mon[27147]: Reconfiguring alertmanager.smithi060 (dependencies changed)... 2024-04-06T03:20:25.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:25 smithi195 ceph-mon[27147]: Reconfiguring daemon alertmanager.smithi060 on smithi060 2024-04-06T03:20:25.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:25 smithi195 ceph-mon[27147]: pgmap v13: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 503 op/s 2024-04-06T03:20:25.607 INFO:teuthology.orchestra.run.smithi060.stdout: 5 85699 12.27 MB/sec execute 29 sec latency 29.478 ms 2024-04-06T03:20:26.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 86444 12.39 MB/sec execute 30 sec latency 29.643 ms 2024-04-06T03:20:27.015 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:26 smithi060 ceph-mon[24896]: pgmap v14: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 801 op/s 2024-04-06T03:20:27.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:26 smithi195 ceph-mon[27147]: pgmap v14: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 801 op/s 2024-04-06T03:20:27.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 87185 12.51 MB/sec execute 31 sec latency 25.445 ms 2024-04-06T03:20:28.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 87927 12.89 MB/sec execute 32 sec latency 31.684 ms 2024-04-06T03:20:29.283 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:29 smithi060 ceph-mon[24896]: pgmap v15: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.6 MiB/s wr, 526 op/s 2024-04-06T03:20:29.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:29 smithi195 ceph-mon[27147]: pgmap v15: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.6 MiB/s wr, 526 op/s 2024-04-06T03:20:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 88510 13.06 MB/sec execute 33 sec latency 38.285 ms 2024-04-06T03:20:30.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 88881 12.94 MB/sec execute 34 sec latency 41.142 ms 2024-04-06T03:20:31.429 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:31 smithi060 ceph-mon[24896]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 557 op/s 2024-04-06T03:20:31.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 89332 12.91 MB/sec execute 35 sec latency 33.199 ms 2024-04-06T03:20:31.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:31 smithi195 ceph-mon[27147]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 557 op/s 2024-04-06T03:20:32.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 89804 12.82 MB/sec execute 36 sec latency 32.187 ms 2024-04-06T03:20:33.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:33 smithi060 ceph-mon[24896]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 890 op/s 2024-04-06T03:20:33.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 90190 12.77 MB/sec execute 37 sec latency 48.028 ms 2024-04-06T03:20:33.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:33 smithi195 ceph-mon[27147]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 890 op/s 2024-04-06T03:20:34.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 90840 12.79 MB/sec execute 38 sec latency 48.634 ms 2024-04-06T03:20:35.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:35 smithi060 ceph-mon[24896]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.0 MiB/s wr, 677 op/s 2024-04-06T03:20:35.237 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:35 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:35.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 91421 12.96 MB/sec execute 39 sec latency 24.544 ms 2024-04-06T03:20:35.654 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:35 smithi195 ceph-mon[27147]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.0 MiB/s wr, 677 op/s 2024-04-06T03:20:35.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:35 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:36.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 91991 13.14 MB/sec execute 40 sec latency 33.910 ms 2024-04-06T03:20:37.009 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:36 smithi060 ceph-mon[24896]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 970 op/s 2024-04-06T03:20:37.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:36 smithi195 ceph-mon[27147]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 970 op/s 2024-04-06T03:20:37.609 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:20:37.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 92441 13.08 MB/sec execute 41 sec latency 32.966 ms 2024-04-06T03:20:38.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 92897 13.05 MB/sec execute 42 sec latency 39.073 ms 2024-04-06T03:20:38.668 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:20:38.668 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (7m) 19s ago 9m 26.5M - 0.23.0 ba2b418f427c 5a135361e3ed 2024-04-06T03:20:38.668 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (9m) 19s ago 9m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (8m) 21s ago 8m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (7m) 19s ago 8m 57.0M - 8.3.5 dad864ee21e9 8160f6c832ff 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (3m) 19s ago 3m 38.8M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (3m) 21s ago 3m 14.1M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (10m) 19s ago 10m 338M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (44s) 21s ago 8m 488M - 19.0.0-2027-g9f192350 a5511c7deba9 3ab7c8547fa3 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (10m) 19s ago 10m 49.3M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (8m) 21s ago 8m 42.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (2m) 19s ago 2m 258M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (2m) 21s ago 2m 73.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (8m) 19s ago 8m 19.7M - 1.3.1 1dbe0e931976 2c4375f28350 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (8m) 21s ago 8m 21.7M - 1.3.1 1dbe0e931976 a0369313f174 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (7m) 19s ago 7m 330M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:20:38.669 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (7m) 19s ago 7m 257M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (6m) 19s ago 6m 197M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (6m) 19s ago 6m 267M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (5m) 21s ago 5m 298M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (5m) 21s ago 5m 319M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (5m) 21s ago 5m 269M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (4m) 21s ago 4m 400M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:20:38.670 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (7m) 19s ago 7m 64.6M - 2.33.4 514e6a882f6e 03d658f74f0b 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:20:39.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:20:39.187 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:20:39.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:39 smithi060 ceph-mon[24896]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:39.334 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:39 smithi060 ceph-mon[24896]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:39.335 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:39 smithi060 ceph-mon[24896]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.7 MiB/s wr, 672 op/s 2024-04-06T03:20:39.335 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:39 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/4201953392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:20:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 93384 13.02 MB/sec execute 43 sec latency 233.182 ms 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T03:20:39.694 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-04-06T03:20:39.695 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:20:39.695 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:20:39.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:39 smithi195 ceph-mon[27147]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:39.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:39 smithi195 ceph-mon[27147]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:39.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:39 smithi195 ceph-mon[27147]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.7 MiB/s wr, 672 op/s 2024-04-06T03:20:39.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:39 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/4201953392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:20:40.346 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:40 smithi060 ceph-mon[24896]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:40.415 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:20:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 93954 13.01 MB/sec execute 44 sec latency 29.853 ms 2024-04-06T03:20:40.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:40 smithi195 ceph-mon[27147]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:41.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:41 smithi060 ceph-mon[24896]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:41.357 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:41 smithi060 ceph-mon[24896]: pgmap v21: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.8 MiB/s wr, 793 op/s 2024-04-06T03:20:41.357 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:41 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/1933705792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:20:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 94479 12.96 MB/sec execute 45 sec latency 40.625 ms 2024-04-06T03:20:41.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:41 smithi195 ceph-mon[27147]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:41.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:41 smithi195 ceph-mon[27147]: pgmap v21: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.8 MiB/s wr, 793 op/s 2024-04-06T03:20:41.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:41 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/1933705792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:20:42.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 94893 13.04 MB/sec execute 46 sec latency 36.298 ms 2024-04-06T03:20:42.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:42.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:42.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: Reconfiguring grafana.smithi060 (dependencies changed)... 2024-04-06T03:20:42.888 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: Reconfiguring daemon grafana.smithi060 on smithi060 2024-04-06T03:20:42.889 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:42 smithi060 ceph-mon[24896]: pgmap v22: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 947 op/s 2024-04-06T03:20:43.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: Reconfiguring grafana.smithi060 (dependencies changed)... 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:20:43.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:20:43.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: Reconfiguring daemon grafana.smithi060 on smithi060 2024-04-06T03:20:43.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:42 smithi195 ceph-mon[27147]: pgmap v22: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 947 op/s 2024-04-06T03:20:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 95384 13.13 MB/sec execute 47 sec latency 30.115 ms 2024-04-06T03:20:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 5 95846 13.11 MB/sec execute 48 sec latency 32.913 ms 2024-04-06T03:20:45.427 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:45 smithi060 ceph-mon[24896]: pgmap v23: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:20:45.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 96178 12.95 MB/sec execute 49 sec latency 56.710 ms 2024-04-06T03:20:45.654 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:45 smithi195 ceph-mon[27147]: pgmap v23: 97 pgs: 97 active+clean; 332 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:20:46.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 96649 12.91 MB/sec execute 50 sec latency 67.540 ms 2024-04-06T03:20:46.943 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:46 smithi060 ceph-mon[24896]: pgmap v24: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 873 op/s 2024-04-06T03:20:47.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:46 smithi195 ceph-mon[27147]: pgmap v24: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 873 op/s 2024-04-06T03:20:47.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 97072 12.89 MB/sec execute 51 sec latency 34.310 ms 2024-04-06T03:20:48.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 97547 12.91 MB/sec execute 52 sec latency 52.933 ms 2024-04-06T03:20:49.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:49 smithi060 ceph-mon[24896]: pgmap v25: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-04-06T03:20:49.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 97929 12.78 MB/sec execute 53 sec latency 98.200 ms 2024-04-06T03:20:49.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:49 smithi195 ceph-mon[27147]: pgmap v25: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-04-06T03:20:50.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:50 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:50.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 98330 12.76 MB/sec execute 54 sec latency 33.897 ms 2024-04-06T03:20:50.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:50 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:20:51.477 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:51 smithi060 ceph-mon[24896]: pgmap v26: 97 pgs: 97 active+clean; 341 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 778 op/s 2024-04-06T03:20:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 98681 12.74 MB/sec execute 55 sec latency 35.178 ms 2024-04-06T03:20:51.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:51 smithi195 ceph-mon[27147]: pgmap v26: 97 pgs: 97 active+clean; 341 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 778 op/s 2024-04-06T03:20:52.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 99181 12.83 MB/sec execute 56 sec latency 44.354 ms 2024-04-06T03:20:53.263 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:53 smithi060 ceph-mon[24896]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 720 op/s 2024-04-06T03:20:53.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 99536 12.77 MB/sec execute 57 sec latency 32.870 ms 2024-04-06T03:20:53.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:53 smithi195 ceph-mon[27147]: pgmap v27: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 720 op/s 2024-04-06T03:20:54.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 99965 12.74 MB/sec execute 58 sec latency 33.080 ms 2024-04-06T03:20:55.299 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:55 smithi060 ceph-mon[24896]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 534 op/s 2024-04-06T03:20:55.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 100446 12.68 MB/sec execute 59 sec latency 44.314 ms 2024-04-06T03:20:55.656 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:55 smithi195 ceph-mon[27147]: pgmap v28: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 534 op/s 2024-04-06T03:20:56.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 100758 12.64 MB/sec execute 60 sec latency 66.854 ms 2024-04-06T03:20:57.150 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:56 smithi060 ceph-mon[24896]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 785 op/s 2024-04-06T03:20:57.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:56 smithi195 ceph-mon[27147]: pgmap v29: 97 pgs: 97 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 785 op/s 2024-04-06T03:20:57.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 101208 12.59 MB/sec execute 61 sec latency 55.337 ms 2024-04-06T03:20:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 101536 12.48 MB/sec execute 62 sec latency 49.506 ms 2024-04-06T03:20:59.422 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:20:59 smithi060 ceph-mon[24896]: pgmap v30: 97 pgs: 97 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 511 op/s 2024-04-06T03:20:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 101964 12.56 MB/sec execute 63 sec latency 30.943 ms 2024-04-06T03:20:59.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:20:59 smithi195 ceph-mon[27147]: pgmap v30: 97 pgs: 97 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 511 op/s 2024-04-06T03:21:00.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 102397 12.60 MB/sec execute 64 sec latency 37.195 ms 2024-04-06T03:21:01.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:01 smithi060 ceph-mon[24896]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 552 op/s 2024-04-06T03:21:01.613 INFO:teuthology.orchestra.run.smithi060.stdout: 5 102791 12.56 MB/sec execute 65 sec latency 46.148 ms 2024-04-06T03:21:01.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:01 smithi195 ceph-mon[27147]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 552 op/s 2024-04-06T03:21:02.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 103093 12.47 MB/sec execute 66 sec latency 46.163 ms 2024-04-06T03:21:03.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:03 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:03.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:03 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:03.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:03 smithi060 ceph-mon[24896]: pgmap v32: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 554 op/s 2024-04-06T03:21:03.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:03 smithi060 ceph-mon[24896]: Reconfiguring node-exporter.smithi060 (dependencies changed)... 2024-04-06T03:21:03.234 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:03 smithi060 ceph-mon[24896]: Reconfiguring daemon node-exporter.smithi060 on smithi060 2024-04-06T03:21:03.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:03 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:03.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:03 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:03.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:03 smithi195 ceph-mon[27147]: pgmap v32: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 554 op/s 2024-04-06T03:21:03.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:03 smithi195 ceph-mon[27147]: Reconfiguring node-exporter.smithi060 (dependencies changed)... 2024-04-06T03:21:03.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:03 smithi195 ceph-mon[27147]: Reconfiguring daemon node-exporter.smithi060 on smithi060 2024-04-06T03:21:03.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 103472 12.44 MB/sec execute 67 sec latency 34.303 ms 2024-04-06T03:21:04.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 103925 12.36 MB/sec execute 68 sec latency 35.979 ms 2024-04-06T03:21:05.262 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:05 smithi060 ceph-mon[24896]: pgmap v33: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.2 MiB/s wr, 491 op/s 2024-04-06T03:21:05.262 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:05.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 104353 12.39 MB/sec execute 69 sec latency 40.327 ms 2024-04-06T03:21:05.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:05 smithi195 ceph-mon[27147]: pgmap v33: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.2 MiB/s wr, 491 op/s 2024-04-06T03:21:05.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:06.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 104863 12.38 MB/sec execute 70 sec latency 42.196 ms 2024-04-06T03:21:07.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:07.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:07.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:06 smithi060 ceph-mon[24896]: Reconfiguring prometheus.smithi060 (dependencies changed)... 2024-04-06T03:21:07.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:06 smithi060 ceph-mon[24896]: Reconfiguring daemon prometheus.smithi060 on smithi060 2024-04-06T03:21:07.030 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:06 smithi060 ceph-mon[24896]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 735 op/s 2024-04-06T03:21:07.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:07.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:07.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:06 smithi195 ceph-mon[27147]: Reconfiguring prometheus.smithi060 (dependencies changed)... 2024-04-06T03:21:07.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:06 smithi195 ceph-mon[27147]: Reconfiguring daemon prometheus.smithi060 on smithi060 2024-04-06T03:21:07.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:06 smithi195 ceph-mon[27147]: pgmap v34: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 735 op/s 2024-04-06T03:21:07.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 105307 12.33 MB/sec execute 71 sec latency 50.390 ms 2024-04-06T03:21:08.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 105746 12.38 MB/sec execute 72 sec latency 36.933 ms 2024-04-06T03:21:09.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:09 smithi060 ceph-mon[24896]: pgmap v35: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s 2024-04-06T03:21:09.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 106209 12.45 MB/sec execute 73 sec latency 52.320 ms 2024-04-06T03:21:09.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:09 smithi195 ceph-mon[27147]: pgmap v35: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s 2024-04-06T03:21:10.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 106486 12.35 MB/sec execute 74 sec latency 35.432 ms 2024-04-06T03:21:11.052 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:21:11.369 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:11 smithi060 ceph-mon[24896]: pgmap v36: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 682 op/s 2024-04-06T03:21:11.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 106669 12.26 MB/sec execute 75 sec latency 42.101 ms 2024-04-06T03:21:11.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:11 smithi195 ceph-mon[27147]: pgmap v36: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 682 op/s 2024-04-06T03:21:12.201 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:21:12.201 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 starting - - - - 2024-04-06T03:21:12.201 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (9m) 53s ago 9m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:21:12.201 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (9m) 55s ago 9m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 starting - - - - 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (3m) 53s ago 3m 38.8M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (3m) 55s ago 3m 14.1M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (10m) 53s ago 10m 338M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (78s) 55s ago 8m 488M - 19.0.0-2027-g9f192350 a5511c7deba9 3ab7c8547fa3 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (11m) 53s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (8m) 55s ago 8m 42.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (3m) 53s ago 3m 258M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (3m) 55s ago 3m 73.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 starting - - - - 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (8m) 55s ago 8m 21.7M - 1.3.1 1dbe0e931976 a0369313f174 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (8m) 53s ago 8m 330M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (7m) 53s ago 7m 257M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:21:12.202 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (7m) 53s ago 7m 197M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (6m) 53s ago 6m 267M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (6m) 55s ago 6m 298M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (6m) 55s ago 6m 319M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (5m) 55s ago 5m 269M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (5m) 55s ago 5m 400M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:21:12.203 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (8m) 53s ago 8m 64.6M - 2.33.4 514e6a882f6e 03d658f74f0b 2024-04-06T03:21:12.398 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:12 smithi060 ceph-mon[24896]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:12.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 106978 12.22 MB/sec execute 76 sec latency 51.689 ms 2024-04-06T03:21:12.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:12 smithi195 ceph-mon[27147]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:21:12.832 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:21:12.833 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:21:13.406 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:21:13.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:13 smithi060 ceph-mon[24896]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:13.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:13 smithi060 ceph-mon[24896]: pgmap v37: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 703 op/s 2024-04-06T03:21:13.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:13 smithi060 ceph-mon[24896]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:13.420 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:13 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3658280937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:21:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 107332 12.14 MB/sec execute 77 sec latency 53.942 ms 2024-04-06T03:21:13.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:13 smithi195 ceph-mon[27147]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:13 smithi195 ceph-mon[27147]: pgmap v37: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 703 op/s 2024-04-06T03:21:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:13 smithi195 ceph-mon[27147]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:13.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:13 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3658280937' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:21:14.082 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:21:14.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:14 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/3605539610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:21:14.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 107833 12.16 MB/sec execute 78 sec latency 59.729 ms 2024-04-06T03:21:14.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:14 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3605539610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:21:15.445 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:15 smithi060 ceph-mon[24896]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:15.445 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:15 smithi060 ceph-mon[24896]: pgmap v38: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-04-06T03:21:15.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 108433 12.18 MB/sec execute 79 sec latency 29.389 ms 2024-04-06T03:21:15.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:15 smithi195 ceph-mon[27147]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:15.620 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:15 smithi195 ceph-mon[27147]: pgmap v38: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-04-06T03:21:16.614 INFO:teuthology.orchestra.run.smithi060.stdout: 5 109090 12.29 MB/sec execute 80 sec latency 31.022 ms 2024-04-06T03:21:16.962 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:16 smithi060 ceph-mon[24896]: pgmap v39: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 723 op/s 2024-04-06T03:21:17.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:16 smithi195 ceph-mon[27147]: pgmap v39: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 723 op/s 2024-04-06T03:21:17.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 109679 12.37 MB/sec execute 81 sec latency 36.994 ms 2024-04-06T03:21:18.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 110143 12.37 MB/sec execute 82 sec latency 52.309 ms 2024-04-06T03:21:19.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:19 smithi060 ceph-mon[24896]: pgmap v40: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 479 op/s 2024-04-06T03:21:19.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 110616 12.35 MB/sec execute 83 sec latency 37.304 ms 2024-04-06T03:21:19.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:19 smithi195 ceph-mon[27147]: pgmap v40: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 479 op/s 2024-04-06T03:21:20.272 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:20 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:20.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 110989 12.28 MB/sec execute 84 sec latency 38.384 ms 2024-04-06T03:21:20.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:20 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:21.287 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:21 smithi060 ceph-mon[24896]: pgmap v41: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-04-06T03:21:21.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 111328 12.28 MB/sec execute 85 sec latency 54.728 ms 2024-04-06T03:21:21.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:21 smithi195 ceph-mon[27147]: pgmap v41: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-04-06T03:21:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 111687 12.22 MB/sec execute 86 sec latency 53.135 ms 2024-04-06T03:21:23.450 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:23 smithi060 ceph-mon[24896]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 603 op/s 2024-04-06T03:21:23.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 112003 12.18 MB/sec execute 87 sec latency 72.797 ms 2024-04-06T03:21:23.718 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:23 smithi195 ceph-mon[27147]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 603 op/s 2024-04-06T03:21:24.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 112293 12.11 MB/sec execute 88 sec latency 54.036 ms 2024-04-06T03:21:25.293 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:25 smithi060 ceph-mon[24896]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 541 op/s 2024-04-06T03:21:25.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 112633 12.12 MB/sec execute 89 sec latency 44.686 ms 2024-04-06T03:21:25.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:25 smithi195 ceph-mon[27147]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 541 op/s 2024-04-06T03:21:26.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 113000 12.12 MB/sec execute 90 sec latency 57.096 ms 2024-04-06T03:21:27.072 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:26 smithi060 ceph-mon[24896]: pgmap v44: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 747 op/s 2024-04-06T03:21:27.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:26 smithi195 ceph-mon[27147]: pgmap v44: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 747 op/s 2024-04-06T03:21:27.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 113452 12.12 MB/sec execute 91 sec latency 32.165 ms 2024-04-06T03:21:28.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 113748 12.07 MB/sec execute 92 sec latency 59.989 ms 2024-04-06T03:21:29.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:29 smithi060 ceph-mon[24896]: pgmap v45: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-04-06T03:21:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 114121 12.04 MB/sec execute 93 sec latency 40.247 ms 2024-04-06T03:21:29.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:29 smithi195 ceph-mon[27147]: pgmap v45: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-04-06T03:21:30.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 114481 11.99 MB/sec execute 94 sec latency 42.063 ms 2024-04-06T03:21:31.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:31 smithi060 ceph-mon[24896]: pgmap v46: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 676 op/s 2024-04-06T03:21:31.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 114860 11.98 MB/sec execute 95 sec latency 36.762 ms 2024-04-06T03:21:31.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:31 smithi195 ceph-mon[27147]: pgmap v46: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 676 op/s 2024-04-06T03:21:32.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 115277 11.95 MB/sec execute 96 sec latency 35.279 ms 2024-04-06T03:21:33.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:33 smithi060 ceph-mon[24896]: pgmap v47: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 599 op/s 2024-04-06T03:21:33.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:33 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:33.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:33 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 115676 11.93 MB/sec execute 97 sec latency 40.140 ms 2024-04-06T03:21:33.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:33 smithi195 ceph-mon[27147]: pgmap v47: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 599 op/s 2024-04-06T03:21:33.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:33 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:33.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:33 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:34 smithi060 ceph-mon[24896]: Reconfiguring node-exporter.smithi195 (dependencies changed)... 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:34 smithi060 ceph-mon[24896]: Reconfiguring daemon node-exporter.smithi195 on smithi195 2024-04-06T03:21:34.501 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:34 smithi195 ceph-mon[27147]: Reconfiguring node-exporter.smithi195 (dependencies changed)... 2024-04-06T03:21:34.501 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:34 smithi195 ceph-mon[27147]: Reconfiguring daemon node-exporter.smithi195 on smithi195 2024-04-06T03:21:34.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 116148 11.97 MB/sec execute 98 sec latency 44.528 ms 2024-04-06T03:21:35.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:35 smithi060 ceph-mon[24896]: pgmap v48: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 425 op/s 2024-04-06T03:21:35.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:35 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:35.424 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:35 smithi195 ceph-mon[27147]: pgmap v48: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 425 op/s 2024-04-06T03:21:35.424 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:35 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:35.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 116561 12.01 MB/sec execute 99 sec latency 48.952 ms 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi060.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi060.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:21:36.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:21:36.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:36 smithi060 ceph-mon[24896]: pgmap v49: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.9 MiB/s wr, 672 op/s 2024-04-06T03:21:36.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 117205 12.04 MB/sec execute 100 sec latency 34.707 ms 2024-04-06T03:21:36.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi060.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi060.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:21:36.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:21:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:36 smithi195 ceph-mon[27147]: pgmap v49: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.9 MiB/s wr, 672 op/s 2024-04-06T03:21:37.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 117598 12.00 MB/sec execute 101 sec latency 52.222 ms 2024-04-06T03:21:37.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:37 smithi060 ceph-mon[24896]: mgrmap e27: smithi195.ultkot(active, since 93s), standbys: smithi060.jttwce 2024-04-06T03:21:37.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:37 smithi195 ceph-mon[27147]: mgrmap e27: smithi195.ultkot(active, since 93s), standbys: smithi060.jttwce 2024-04-06T03:21:38.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 118042 11.97 MB/sec execute 102 sec latency 50.053 ms 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 466 op/s 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:21:38.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:38 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 466 op/s 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:21:38.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:38 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 118475 11.99 MB/sec execute 103 sec latency 34.794 ms 2024-04-06T03:21:39.779 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:39 smithi060 ceph-mon[24896]: Upgrade: Updating mgr.smithi060.jttwce 2024-04-06T03:21:39.779 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:39 smithi060 ceph-mon[24896]: Deploying daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:21:39.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:39 smithi195 ceph-mon[27147]: Upgrade: Updating mgr.smithi060.jttwce 2024-04-06T03:21:39.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:39 smithi195 ceph-mon[27147]: Deploying daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:21:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 118878 11.97 MB/sec execute 104 sec latency 38.739 ms 2024-04-06T03:21:40.805 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:40 smithi060 ceph-mon[24896]: pgmap v51: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 693 op/s 2024-04-06T03:21:40.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:40 smithi195 ceph-mon[27147]: pgmap v51: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 693 op/s 2024-04-06T03:21:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: 5 119179 11.93 MB/sec execute 105 sec latency 54.984 ms 2024-04-06T03:21:42.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 119704 11.97 MB/sec execute 106 sec latency 37.519 ms 2024-04-06T03:21:43.328 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:43 smithi060 ceph-mon[24896]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-04-06T03:21:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 120579 12.11 MB/sec execute 107 sec latency 18.841 ms 2024-04-06T03:21:43.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:43 smithi195 ceph-mon[27147]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-04-06T03:21:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 121150 12.10 MB/sec execute 108 sec latency 37.418 ms 2024-04-06T03:21:44.648 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:21:45.338 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:45 smithi060 ceph-mon[24896]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-06T03:21:45.521 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:45 smithi195 ceph-mon[27147]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-06T03:21:45.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 121671 12.12 MB/sec execute 109 sec latency 30.815 ms 2024-04-06T03:21:45.794 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:21:45.794 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 starting - - - - 2024-04-06T03:21:45.794 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (10m) 86s ago 10m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:21:45.794 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (9m) 88s ago 9m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:21:45.794 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 starting - - - - 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (4m) 86s ago 4m 38.8M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (4m) 88s ago 4m 14.1M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:9283 running (11m) 86s ago 11m 338M - 17.2.0 e1d6a67b021e cf27efa4b500 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (111s) 88s ago 9m 488M - 19.0.0-2027-g9f192350 a5511c7deba9 3ab7c8547fa3 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (11m) 86s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (9m) 88s ago 9m 42.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (4m) 86s ago 4m 258M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (3m) 88s ago 3m 73.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 starting - - - - 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 starting - - - - 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (8m) 86s ago 8m 330M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (8m) 86s ago 8m 257M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (7m) 86s ago 7m 197M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:21:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (7m) 86s ago 7m 267M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:21:45.796 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (7m) 88s ago 7m 298M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:21:45.796 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (6m) 88s ago 6m 319M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:21:45.796 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (6m) 88s ago 6m 269M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:21:45.796 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (5m) 88s ago 5m 400M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:21:45.796 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 starting - - - - 2024-04-06T03:21:46.366 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:46 smithi060 ceph-mon[24896]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:46.366 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:46 smithi060 ceph-mon[24896]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:21:46.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:21:46.398 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:21:46.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 122170 12.13 MB/sec execute 110 sec latency 65.018 ms 2024-04-06T03:21:46.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:46 smithi195 ceph-mon[27147]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:46.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:46 smithi195 ceph-mon[27147]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading mgr daemons", 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:21:46.916 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:21:47.394 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:47 smithi060 ceph-mon[24896]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:47.394 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:47 smithi060 ceph-mon[24896]: pgmap v54: 97 pgs: 97 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 859 op/s 2024-04-06T03:21:47.395 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:47 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/461451125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:21:47.495 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:21:47.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 122608 12.13 MB/sec execute 111 sec latency 26.553 ms 2024-04-06T03:21:47.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:47 smithi195 ceph-mon[27147]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:47.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:47 smithi195 ceph-mon[27147]: pgmap v54: 97 pgs: 97 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 859 op/s 2024-04-06T03:21:47.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:47 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/461451125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:21:48.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:48 smithi060 ceph-mon[24896]: from='client.24531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:48.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:48 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/4049164578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:21:48.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:48 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:48.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:48 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:48.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:48 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:48.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 123007 12.13 MB/sec execute 112 sec latency 39.112 ms 2024-04-06T03:21:48.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:48 smithi195 ceph-mon[27147]: from='client.24531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:48.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:48 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/4049164578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:21:48.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:48 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:48.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:48 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:48.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:48 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:49.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:49 smithi195 ceph-mon[27147]: pgmap v55: 97 pgs: 97 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.2 MiB/s wr, 613 op/s 2024-04-06T03:21:49.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:49 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:49.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:49 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:49.470 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:49 smithi060 ceph-mon[24896]: pgmap v55: 97 pgs: 97 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.2 MiB/s wr, 613 op/s 2024-04-06T03:21:49.470 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:49 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:49.470 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:49 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:49.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 123500 12.15 MB/sec execute 113 sec latency 28.521 ms 2024-04-06T03:21:50.476 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:50 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:50.616 INFO:teuthology.orchestra.run.smithi060.stdout: 5 124048 12.19 MB/sec execute 114 sec latency 31.548 ms 2024-04-06T03:21:50.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:50 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:51.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:51 smithi060 ceph-mon[24896]: pgmap v56: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 779 op/s 2024-04-06T03:21:51.577 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:51 smithi195 ceph-mon[27147]: pgmap v56: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 779 op/s 2024-04-06T03:21:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: 5 124492 12.15 MB/sec execute 115 sec latency 60.850 ms 2024-04-06T03:21:52.617 INFO:teuthology.orchestra.run.smithi060.stdout: 5 124807 12.11 MB/sec execute 116 sec latency 75.739 ms 2024-04-06T03:21:53.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:53 smithi060 ceph-mon[24896]: pgmap v57: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 7.8 MiB/s wr, 668 op/s 2024-04-06T03:21:53.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:53 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:53.617 INFO:teuthology.orchestra.run.smithi060.stdout: 5 125189 12.10 MB/sec execute 117 sec latency 45.316 ms 2024-04-06T03:21:53.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:53 smithi195 ceph-mon[27147]: pgmap v57: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 7.8 MiB/s wr, 668 op/s 2024-04-06T03:21:53.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:53 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:54.263 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:54 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:54.617 INFO:teuthology.orchestra.run.smithi060.stdout: 5 125876 12.15 MB/sec execute 118 sec latency 30.798 ms 2024-04-06T03:21:54.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:54 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:55.270 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:55 smithi060 ceph-mon[24896]: pgmap v58: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 605 op/s 2024-04-06T03:21:55.620 INFO:teuthology.orchestra.run.smithi060.stdout: 5 126372 12.16 MB/sec execute 119 sec latency 32.555 ms 2024-04-06T03:21:55.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:55 smithi195 ceph-mon[27147]: pgmap v58: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 605 op/s 2024-04-06T03:21:56.620 INFO:teuthology.orchestra.run.smithi060.stdout: 5 126794 12.17 MB/sec execute 120 sec latency 68.918 ms 2024-04-06T03:21:57.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:56 smithi060 ceph-mon[24896]: pgmap v59: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 889 op/s 2024-04-06T03:21:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:56 smithi195 ceph-mon[27147]: pgmap v59: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 889 op/s 2024-04-06T03:21:57.620 INFO:teuthology.orchestra.run.smithi060.stdout: 5 127130 12.14 MB/sec execute 121 sec latency 56.551 ms 2024-04-06T03:21:58.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: Standby manager daemon smithi060.jttwce restarted 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: Standby manager daemon smithi060.jttwce started 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/crt"}]: dispatch 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/key"}]: dispatch 2024-04-06T03:21:58.324 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:58 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:21:58.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:58.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:21:58.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: Standby manager daemon smithi060.jttwce restarted 2024-04-06T03:21:58.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: Standby manager daemon smithi060.jttwce started 2024-04-06T03:21:58.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/crt"}]: dispatch 2024-04-06T03:21:58.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:21:58.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi060.jttwce/key"}]: dispatch 2024-04-06T03:21:58.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:58 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:21:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: 5 127459 12.16 MB/sec execute 122 sec latency 61.618 ms 2024-04-06T03:21:59.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:59 smithi060 ceph-mon[24896]: mgrmap e28: smithi195.ultkot(active, since 114s), standbys: smithi060.jttwce 2024-04-06T03:21:59.353 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:21:59 smithi060 ceph-mon[24896]: pgmap v60: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-04-06T03:21:59.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:59 smithi195 ceph-mon[27147]: mgrmap e28: smithi195.ultkot(active, since 114s), standbys: smithi060.jttwce 2024-04-06T03:21:59.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:21:59 smithi195 ceph-mon[27147]: pgmap v60: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 565 op/s 2024-04-06T03:21:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: 5 127763 12.10 MB/sec execute 123 sec latency 43.987 ms 2024-04-06T03:22:00.621 INFO:teuthology.orchestra.run.smithi060.stdout: 5 128090 12.06 MB/sec execute 124 sec latency 54.267 ms 2024-04-06T03:22:00.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:00 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:00.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:00 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:00.892 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:00 smithi060 ceph-mon[24896]: pgmap v61: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-04-06T03:22:01.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:00 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:01.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:00 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:01.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:00 smithi195 ceph-mon[27147]: pgmap v61: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-04-06T03:22:01.621 INFO:teuthology.orchestra.run.smithi060.stdout: 5 128362 11.99 MB/sec execute 125 sec latency 48.450 ms 2024-04-06T03:22:02.621 INFO:teuthology.orchestra.run.smithi060.stdout: 5 128665 11.97 MB/sec execute 126 sec latency 47.296 ms 2024-04-06T03:22:03.468 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:03 smithi060 ceph-mon[24896]: pgmap v62: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 616 op/s 2024-04-06T03:22:03.621 INFO:teuthology.orchestra.run.smithi060.stdout: 5 129035 11.96 MB/sec execute 127 sec latency 71.789 ms 2024-04-06T03:22:03.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:03 smithi195 ceph-mon[27147]: pgmap v62: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.7 MiB/s wr, 616 op/s 2024-04-06T03:22:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 129447 11.95 MB/sec execute 128 sec latency 45.384 ms 2024-04-06T03:22:05.507 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:05 smithi060 ceph-mon[24896]: pgmap v63: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 501 op/s 2024-04-06T03:22:05.507 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:05 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:05.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 129796 11.95 MB/sec execute 129 sec latency 35.951 ms 2024-04-06T03:22:05.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:05 smithi195 ceph-mon[27147]: pgmap v63: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 501 op/s 2024-04-06T03:22:05.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:05 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:06.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 130292 11.95 MB/sec execute 130 sec latency 58.641 ms 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: pgmap v64: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 742 op/s 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:22:06.794 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:22:06.795 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: Failing over to other MGR 2024-04-06T03:22:06.795 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:06.795 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:06.795 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:22:06.795 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:06 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: pgmap v64: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 742 op/s 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.956 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: Upgrade: Need to upgrade myself (mgr.smithi195.ultkot) 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: Failing over to other MGR 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 172.21.15.195:0/3487232057' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:22:06.957 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:06 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:07.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 130657 11.94 MB/sec execute 131 sec latency 38.423 ms 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd='[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]': finished 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: mgrmap e29: smithi060.jttwce(active, starting, since 0.866045s), standbys: smithi195.ultkot 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:22:07.826 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:22:07.827 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:07.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:07.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:07.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:22:07.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:22:07.828 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:07 smithi060 ceph-mon[24896]: [06/Apr/2024:03:22:07] ENGINE Bus STARTING 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.24433 ' entity='mgr.smithi195.ultkot' cmd='[{"prefix": "mgr fail", "who": "smithi195.ultkot"}]': finished 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: mgrmap e29: smithi060.jttwce(active, starting, since 0.866045s), standbys: smithi195.ultkot 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:22:07.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:22:07.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:22:07.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:22:07.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:07 smithi195 ceph-mon[27147]: [06/Apr/2024:03:22:07] ENGINE Bus STARTING 2024-04-06T03:22:08.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 130978 11.94 MB/sec execute 132 sec latency 34.995 ms 2024-04-06T03:22:08.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: [06/Apr/2024:03:22:07] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:22:08.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: [06/Apr/2024:03:22:07] ENGINE Serving on http://172.21.15.60:8765 2024-04-06T03:22:08.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: [06/Apr/2024:03:22:07] ENGINE Bus STARTED 2024-04-06T03:22:08.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: mgrmap e30: smithi060.jttwce(active, since 1.87204s), standbys: smithi195.ultkot 2024-04-06T03:22:08.819 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: pgmap v3: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:08.820 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:08.820 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:08 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: [06/Apr/2024:03:22:07] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: [06/Apr/2024:03:22:07] ENGINE Serving on http://172.21.15.60:8765 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: [06/Apr/2024:03:22:07] ENGINE Bus STARTED 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: mgrmap e30: smithi060.jttwce(active, since 1.87204s), standbys: smithi195.ultkot 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: pgmap v3: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:08.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:08 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:09.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 131240 11.90 MB/sec execute 133 sec latency 67.253 ms 2024-04-06T03:22:09.859 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:09 smithi195 ceph-mon[27147]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:22:09.859 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:09 smithi195 ceph-mon[27147]: Cluster is now healthy 2024-04-06T03:22:09.859 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:09 smithi195 ceph-mon[27147]: pgmap v4: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:09.862 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:09 smithi060 ceph-mon[24896]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:22:09.862 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:09 smithi060 ceph-mon[24896]: Cluster is now healthy 2024-04-06T03:22:09.862 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:09 smithi060 ceph-mon[24896]: pgmap v4: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:10.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 131530 11.87 MB/sec execute 134 sec latency 58.998 ms 2024-04-06T03:22:10.879 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:10 smithi060 ceph-mon[24896]: mgrmap e31: smithi060.jttwce(active, since 3s), standbys: smithi195.ultkot 2024-04-06T03:22:10.969 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:10 smithi195 ceph-mon[27147]: mgrmap e31: smithi060.jttwce(active, since 3s), standbys: smithi195.ultkot 2024-04-06T03:22:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 131772 11.81 MB/sec execute 135 sec latency 72.574 ms 2024-04-06T03:22:12.150 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:11 smithi060 ceph-mon[24896]: pgmap v5: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:12.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:11 smithi195 ceph-mon[27147]: pgmap v5: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:12.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 132024 11.75 MB/sec execute 136 sec latency 51.037 ms 2024-04-06T03:22:13.477 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:13 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:13.477 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:13 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:13.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 132308 11.73 MB/sec execute 137 sec latency 52.298 ms 2024-04-06T03:22:13.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:13 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:13.655 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:13 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:14.507 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:14 smithi060 ceph-mon[24896]: pgmap v6: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:14.507 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:14 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:14.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:14 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:14.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 132708 11.73 MB/sec execute 138 sec latency 56.551 ms 2024-04-06T03:22:14.717 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:14 smithi195 ceph-mon[27147]: pgmap v6: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:22:14.717 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:14 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:14.717 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:14 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:15.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 133063 11.70 MB/sec execute 139 sec latency 49.046 ms 2024-04-06T03:22:16.540 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 321 op/s 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:22:16.541 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:16 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:22:16.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 133386 11.70 MB/sec execute 140 sec latency 83.184 ms 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 321 op/s 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:22:16.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:16 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/178308431' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:22:17.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:17 smithi195 ceph-mon[27147]: mgrmap e32: smithi060.jttwce(active, since 10s), standbys: smithi195.ultkot 2024-04-06T03:22:17.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:17 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:17.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:17 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:17.576 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:17 smithi060 ceph-mon[24896]: mgrmap e32: smithi060.jttwce(active, since 10s), standbys: smithi195.ultkot 2024-04-06T03:22:17.577 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:17 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:17.577 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:17 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:17.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 133825 11.69 MB/sec execute 141 sec latency 46.926 ms 2024-04-06T03:22:18.057 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:22:18.601 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:18 smithi060 ceph-mon[24896]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-04-06T03:22:18.602 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:18 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:18.602 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:18 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:18.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 134126 11.67 MB/sec execute 142 sec latency 39.230 ms 2024-04-06T03:22:18.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:18 smithi195 ceph-mon[27147]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-04-06T03:22:18.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:18 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:18.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:18 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:19.008 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (98s) 2s ago 10m 19.8M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (10m) 2s ago 10m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (10m) 6s ago 10m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 unknown 2s ago 10m - - 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (4m) 2s ago 4m 39.6M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (4m) 6s ago 4m 15.0M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (32s) 2s ago 12m 482M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283 running (2m) 6s ago 10m 112M - 19.0.0-2027-g9f192350 a5511c7deba9 3ab7c8547fa3 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (12m) 2s ago 12m 55.1M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (10m) 6s ago 10m 46.7M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (4m) 2s ago 4m 275M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:22:19.009 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (4m) 6s ago 4m 79.4M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (73s) 2s ago 10m 11.5M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (44s) 6s ago 9m 8581k - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (9m) 2s ago 9m 478M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (8m) 2s ago 8m 411M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (8m) 2s ago 8m 286M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (7m) 2s ago 7m 435M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (7m) 6s ago 7m 454M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (7m) 6s ago 7m 492M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (6m) 6s ago 6m 417M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (6m) 6s ago 6m 356M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:22:19.010 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (46s) 2s ago 9m 53.7M - 2.33.4 514e6a882f6e c80c50394cdb 2024-04-06T03:22:19.523 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:19.524 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:22:19.525 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-06T03:22:19.525 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:22:19.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:22:19.525 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:22:19.598 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:19 smithi060 ceph-mon[24896]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:19.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 134415 11.67 MB/sec execute 143 sec latency 54.541 ms 2024-04-06T03:22:19.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:19 smithi195 ceph-mon[27147]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr" 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:22:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "2/23 daemons upgraded", 2024-04-06T03:22:19.921 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-04-06T03:22:19.921 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:22:19.921 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:22:20.436 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:22:20.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 134638 11.64 MB/sec execute 144 sec latency 68.661 ms 2024-04-06T03:22:20.623 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:20 smithi060 ceph-mon[24896]: from='client.24567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:20.623 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:20 smithi060 ceph-mon[24896]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.1 MiB/s wr, 432 op/s 2024-04-06T03:22:20.623 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:20 smithi060 ceph-mon[24896]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:20.623 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:20 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2072211112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:20.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:20 smithi195 ceph-mon[27147]: from='client.24567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:20.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:20 smithi195 ceph-mon[27147]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.1 MiB/s wr, 432 op/s 2024-04-06T03:22:20.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:20 smithi195 ceph-mon[27147]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:20.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:20 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2072211112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:21.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 134917 11.58 MB/sec execute 145 sec latency 58.872 ms 2024-04-06T03:22:21.636 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:21 smithi060 ceph-mon[24896]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:21.636 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:21 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2648226589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:22:21.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:21 smithi195 ceph-mon[27147]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:21.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:21 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2648226589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:22:22.402 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:22 smithi060 ceph-mon[24896]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 395 op/s 2024-04-06T03:22:22.402 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:22 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:22.402 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:22 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: 5 135215 11.56 MB/sec execute 146 sec latency 65.054 ms 2024-04-06T03:22:22.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:22 smithi195 ceph-mon[27147]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 395 op/s 2024-04-06T03:22:22.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:22 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:22.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:22 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:23.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 135606 11.52 MB/sec execute 147 sec latency 67.600 ms 2024-04-06T03:22:24.456 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:24 smithi060 ceph-mon[24896]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 395 op/s 2024-04-06T03:22:24.456 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:24 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:24.456 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:24 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:24.456 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:24 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:24.456 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:24 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:24.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 136042 11.52 MB/sec execute 148 sec latency 52.975 ms 2024-04-06T03:22:24.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:24 smithi195 ceph-mon[27147]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 395 op/s 2024-04-06T03:22:24.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:24 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:24.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:24 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:24.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:24 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:24.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:24 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:25.487 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.488 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:25 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 136522 11.55 MB/sec execute 149 sec latency 33.374 ms 2024-04-06T03:22:25.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:25 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:26.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:22:26.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:22:26.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.6 MiB/s wr, 571 op/s 2024-04-06T03:22:26.513 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: pgmap v13: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-06T03:22:26.513 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: Reconfiguring prometheus.smithi060 (dependencies changed)... 2024-04-06T03:22:26.513 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: Reconfiguring daemon prometheus.smithi060 on smithi060 2024-04-06T03:22:26.513 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:26 smithi060 ceph-mon[24896]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:22:26.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 136890 11.55 MB/sec execute 150 sec latency 40.982 ms 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.6 MiB/s wr, 571 op/s 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: pgmap v13: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: Reconfiguring prometheus.smithi060 (dependencies changed)... 2024-04-06T03:22:26.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: Reconfiguring daemon prometheus.smithi060 on smithi060 2024-04-06T03:22:26.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:26 smithi195 ceph-mon[27147]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:22:27.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 137330 11.55 MB/sec execute 151 sec latency 30.349 ms 2024-04-06T03:22:28.536 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:28 smithi060 ceph-mon[24896]: pgmap v14: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-06T03:22:28.624 INFO:teuthology.orchestra.run.smithi060.stdout: 5 137728 11.54 MB/sec execute 152 sec latency 33.084 ms 2024-04-06T03:22:28.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:28 smithi195 ceph-mon[27147]: pgmap v14: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-06T03:22:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: 5 138070 11.54 MB/sec execute 153 sec latency 36.498 ms 2024-04-06T03:22:30.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:30 smithi060 ceph-mon[24896]: pgmap v15: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-06T03:22:30.625 INFO:teuthology.orchestra.run.smithi060.stdout: 5 138407 11.52 MB/sec execute 154 sec latency 55.159 ms 2024-04-06T03:22:30.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:30 smithi195 ceph-mon[27147]: pgmap v15: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-06T03:22:31.625 INFO:teuthology.orchestra.run.smithi060.stdout: 5 138843 11.51 MB/sec execute 155 sec latency 74.383 ms 2024-04-06T03:22:32.593 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:32 smithi060 ceph-mon[24896]: pgmap v16: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-06T03:22:32.625 INFO:teuthology.orchestra.run.smithi060.stdout: 5 139363 11.52 MB/sec execute 156 sec latency 33.862 ms 2024-04-06T03:22:32.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:32 smithi195 ceph-mon[27147]: pgmap v16: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 508 op/s 2024-04-06T03:22:33.625 INFO:teuthology.orchestra.run.smithi060.stdout: 5 139942 11.54 MB/sec execute 157 sec latency 38.380 ms 2024-04-06T03:22:34.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:34 smithi060 ceph-mon[24896]: pgmap v17: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 806 op/s 2024-04-06T03:22:34.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:34 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:34.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:34 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:34.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:34 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:34.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:34 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:34.626 INFO:teuthology.orchestra.run.smithi060.stdout: 5 140521 11.59 MB/sec execute 158 sec latency 33.599 ms 2024-04-06T03:22:34.673 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:34 smithi195 ceph-mon[27147]: pgmap v17: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 806 op/s 2024-04-06T03:22:34.673 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:34 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:34.673 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:34 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:34.673 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:34 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:34.673 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:34 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:35.626 INFO:teuthology.orchestra.run.smithi060.stdout: 5 141011 11.59 MB/sec execute 159 sec latency 28.972 ms 2024-04-06T03:22:35.647 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:35 smithi060 ceph-mon[24896]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:35.676 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:35 smithi195 ceph-mon[27147]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:36.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 141553 11.63 MB/sec execute 160 sec latency 34.181 ms 2024-04-06T03:22:36.660 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:36 smithi060 ceph-mon[24896]: pgmap v18: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:36.660 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:36 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:22:36.660 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:36 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:22:36.660 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:36 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:36.710 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:36 smithi195 ceph-mon[27147]: pgmap v18: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:36.710 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:36 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:22:36.710 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:36 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:22:36.710 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:36 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:37.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 142052 11.64 MB/sec execute 161 sec latency 55.874 ms 2024-04-06T03:22:37.666 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:37 smithi060 ceph-mon[24896]: Upgrade: Updating mgr.smithi195.ultkot 2024-04-06T03:22:37.667 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:37 smithi060 ceph-mon[24896]: Deploying daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:22:37.667 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:37 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:37.667 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:37 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:37.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:37 smithi195 ceph-mon[27147]: Upgrade: Updating mgr.smithi195.ultkot 2024-04-06T03:22:37.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:37 smithi195 ceph-mon[27147]: Deploying daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:22:37.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:37 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:37.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:37 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:38.422 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:38 smithi060 ceph-mon[24896]: pgmap v19: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 509 op/s 2024-04-06T03:22:38.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 142569 11.62 MB/sec execute 162 sec latency 45.085 ms 2024-04-06T03:22:38.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:38 smithi195 ceph-mon[27147]: pgmap v19: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 509 op/s 2024-04-06T03:22:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 142951 11.62 MB/sec execute 163 sec latency 71.868 ms 2024-04-06T03:22:40.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:40 smithi060 ceph-mon[24896]: pgmap v20: 97 pgs: 97 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-04-06T03:22:40.517 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:40 smithi195 ceph-mon[27147]: pgmap v20: 97 pgs: 97 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-04-06T03:22:40.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 143530 11.65 MB/sec execute 164 sec latency 31.166 ms 2024-04-06T03:22:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 143958 11.66 MB/sec execute 165 sec latency 59.196 ms 2024-04-06T03:22:41.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:41 smithi060 ceph-mon[24896]: pgmap v21: 97 pgs: 97 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.9 MiB/s wr, 575 op/s 2024-04-06T03:22:41.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:41 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:41.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:41 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:41.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:41 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:41 smithi195 ceph-mon[27147]: pgmap v21: 97 pgs: 97 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.9 MiB/s wr, 575 op/s 2024-04-06T03:22:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:41 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:41 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:41 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:42.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 144344 11.64 MB/sec execute 166 sec latency 45.739 ms 2024-04-06T03:22:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 144828 11.65 MB/sec execute 167 sec latency 27.229 ms 2024-04-06T03:22:43.996 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:43 smithi060 ceph-mon[24896]: pgmap v22: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 853 op/s 2024-04-06T03:22:44.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:43 smithi195 ceph-mon[27147]: pgmap v22: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 853 op/s 2024-04-06T03:22:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 145245 11.68 MB/sec execute 168 sec latency 24.321 ms 2024-04-06T03:22:45.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 145679 11.67 MB/sec execute 169 sec latency 67.116 ms 2024-04-06T03:22:46.196 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:45 smithi060 ceph-mon[24896]: pgmap v23: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:46.196 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:45 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:46.196 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:45 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:46.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:45 smithi195 ceph-mon[27147]: pgmap v23: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:46.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:45 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:46.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:45 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:46.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 146242 11.67 MB/sec execute 170 sec latency 33.900 ms 2024-04-06T03:22:47.627 INFO:teuthology.orchestra.run.smithi060.stdout: 5 146663 11.65 MB/sec execute 171 sec latency 44.389 ms 2024-04-06T03:22:48.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:48 smithi060 ceph-mon[24896]: pgmap v24: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:48.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:48 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:48.246 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:48 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:48.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:48 smithi195 ceph-mon[27147]: pgmap v24: 97 pgs: 97 active+clean; 348 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 599 op/s 2024-04-06T03:22:48.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:48 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:48.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:48 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:48.630 INFO:teuthology.orchestra.run.smithi060.stdout: 5 146987 11.64 MB/sec execute 172 sec latency 58.821 ms 2024-04-06T03:22:49.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 147302 11.62 MB/sec execute 173 sec latency 99.127 ms 2024-04-06T03:22:50.056 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: pgmap v25: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 886 op/s 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:22:50.057 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:49 smithi060 ceph-mon[24896]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: pgmap v25: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 886 op/s 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:22:50.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:22:50.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:22:50.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:49 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/1660314358' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:22:50.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 147654 11.63 MB/sec execute 174 sec latency 45.107 ms 2024-04-06T03:22:50.924 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:22:51.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:51 smithi060 ceph-mon[24896]: mgrmap e33: smithi060.jttwce(active, since 44s), standbys: smithi195.ultkot 2024-04-06T03:22:51.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:51 smithi195 ceph-mon[27147]: mgrmap e33: smithi060.jttwce(active, since 44s), standbys: smithi195.ultkot 2024-04-06T03:22:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 148020 11.61 MB/sec execute 175 sec latency 46.902 ms 2024-04-06T03:22:52.001 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (2m) 3s ago 11m 19.8M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (11m) 3s ago 11m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (10m) 6s ago 10m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 unknown 3s ago 11m - - 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (5m) 3s ago 5m 40.2M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (5m) 6s ago 5m 15.1M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (65s) 3s ago 12m 497M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (11s) 6s ago 10m 66.5M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (12m) 3s ago 12m 55.5M 2048M 17.2.0 e1d6a67b021e 453014fdd14a 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (10m) 6s ago 10m 47.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (5m) 3s ago 5m 268M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (5m) 6s ago 5m 80.9M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (106s) 3s ago 11m 12.7M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:22:52.002 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (77s) 6s ago 10m 18.8M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (9m) 3s ago 9m 509M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (9m) 3s ago 9m 439M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (8m) 3s ago 8m 320M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (8m) 3s ago 8m 463M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (8m) 6s ago 8m 481M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (7m) 6s ago 7m 555M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (7m) 6s ago 7m 466M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (6m) 6s ago 6m 400M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:22:52.003 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (19s) 3s ago 10m 46.8M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: pgmap v26: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 564 op/s 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:52.355 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: pgmap v27: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 653 op/s 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: pgmap v28: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: Upgrade: Setting container_image for all mgr 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi060.jttwce"}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi060.jttwce"}]': finished 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi195.ultkot"}]: dispatch 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi195.ultkot"}]': finished 2024-04-06T03:22:52.356 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T03:22:52.357 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:52 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 148391 11.61 MB/sec execute 176 sec latency 36.861 ms 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:22:52.678 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: pgmap v26: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 564 op/s 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: pgmap v27: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 653 op/s 2024-04-06T03:22:52.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: pgmap v28: 97 pgs: 97 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: Upgrade: Setting container_image for all mgr 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi060.jttwce"}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi060.jttwce"}]': finished 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi195.ultkot"}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi195.ultkot"}]': finished 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T03:22:52.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:52 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:53.280 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr" 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "2/23 daemons upgraded", 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:22:53.281 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:22:53.367 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:53 smithi060 ceph-mon[24896]: from='client.24597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:53.367 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:53 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/2225725822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:53.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 148786 11.61 MB/sec execute 177 sec latency 26.274 ms 2024-04-06T03:22:53.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:53 smithi195 ceph-mon[27147]: from='client.24597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:53.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:53 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2225725822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:22:53.965 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:22:53.965 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:22:53.965 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: pgmap v29: 97 pgs: 97 active+clean; 350 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 710 op/s 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='client.? 172.21.15.60:0/261056920' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:22:54.378 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:54 smithi060 ceph-mon[24896]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:54.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 149266 11.62 MB/sec execute 178 sec latency 36.942 ms 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: pgmap v29: 97 pgs: 97 active+clean; 350 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 710 op/s 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/261056920' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:22:54.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:54 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:55.385 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:55 smithi060 ceph-mon[24896]: Upgrade: Updating mon.smithi060 2024-04-06T03:22:55.385 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:55 smithi060 ceph-mon[24896]: Deploying daemon mon.smithi060 on smithi060 2024-04-06T03:22:55.469 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:55 smithi195 ceph-mon[27147]: Upgrade: Updating mon.smithi060 2024-04-06T03:22:55.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:55 smithi195 ceph-mon[27147]: Deploying daemon mon.smithi060 on smithi060 2024-04-06T03:22:55.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 149702 11.59 MB/sec execute 179 sec latency 47.990 ms 2024-04-06T03:22:56.400 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:56 smithi060 ceph-mon[24896]: pgmap v30: 97 pgs: 97 active+clean; 350 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 710 op/s 2024-04-06T03:22:56.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 150112 11.59 MB/sec execute 180 sec latency 47.163 ms 2024-04-06T03:22:56.719 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:22:56 smithi195 ceph-mon[27147]: pgmap v30: 97 pgs: 97 active+clean; 350 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 710 op/s 2024-04-06T03:22:57.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 150541 11.59 MB/sec execute 181 sec latency 41.583 ms 2024-04-06T03:22:57.695 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:57 smithi060 systemd[1]: Stopping Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:22:58.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:58 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[24892]: 2024-04-06T03:22:58.063+0000 7f2837587700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi060 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:22:58.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:58 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[24892]: 2024-04-06T03:22:58.064+0000 7f2837587700 -1 mon.smithi060@0(leader) e2 *** Got Signal Terminated *** 2024-04-06T03:22:58.203 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:58 smithi060 podman[66951]: 2024-04-06 03:22:58.146426422 +0000 UTC m=+0.251018931 container died 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., architecture=x86_64, io.buildah.version=1.19.8, name=centos-stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container) 2024-04-06T03:22:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 150927 11.58 MB/sec execute 182 sec latency 38.590 ms 2024-04-06T03:22:58.964 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:58 smithi060 podman[66951]: 2024-04-06 03:22:58.937926384 +0000 UTC m=+1.042518899 container cleanup 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_BRANCH=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T03:22:58.965 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:58 smithi060 bash[66951]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060 2024-04-06T03:22:59.466 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:59 smithi060 podman[66974]: 2024-04-06 03:22:59.261163758 +0000 UTC m=+1.116156442 container remove 453014fdd14aa0c34dca7a4ecea682a2156a17990134ff267f6d924e6f989fdd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc.) 2024-04-06T03:22:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 151308 11.60 MB/sec execute 183 sec latency 48.933 ms 2024-04-06T03:22:59.724 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:59 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service: Deactivated successfully. 2024-04-06T03:22:59.724 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:59 smithi060 systemd[1]: Stopped Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:59.724 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:59 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service: Consumed 16.973s CPU time. 2024-04-06T03:22:59.975 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:22:59 smithi060 systemd[1]: Starting Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:23:00.226 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 podman[67098]: 2024-04-06 03:23:00.068542596 +0000 UTC m=+0.026564621 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:23:00.226 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 podman[67098]: 2024-04-06 03:23:00.192610193 +0000 UTC m=+0.150632206 container create f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-06T03:23:00.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 151709 11.60 MB/sec execute 184 sec latency 37.127 ms 2024-04-06T03:23:00.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 podman[67098]: 2024-04-06 03:23:00.513650572 +0000 UTC m=+0.471672589 container init f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235) 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 podman[67098]: 2024-04-06 03:23:00.51986359 +0000 UTC m=+0.477885596 container start f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 2 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: pidfile_write: ignore empty --pid-file 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: load: jerasure load: lrc 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: RocksDB version: 7.9.2 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Git sha 0 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: DB SUMMARY 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: DB Session ID: JGTW2PY65OQAP7HFESKZ 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: CURRENT file: CURRENT 2024-04-06T03:23:00.737 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi060/store.db dir, Total Num: 1, files: 000030.sst 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi060/store.db: 000028.log size: 2928913 ; 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.error_if_exists: 0 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.create_if_missing: 0 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.paranoid_checks: 1 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.env: 0x55b66789ec60 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:23:00.738 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.info_log: 0x55b6682c39a0 2024-04-06T03:23:00.739 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:23:00.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.statistics: (nil) 2024-04-06T03:23:00.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.use_fsync: 0 2024-04-06T03:23:00.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_log_file_size: 0 2024-04-06T03:23:00.740 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_fallocate: 1 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.use_direct_reads: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.db_log_dir: 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.wal_dir: 2024-04-06T03:23:00.741 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.write_buffer_manager: 0x55b6682961e0 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:23:00.742 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.unordered_write: 0 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.row_cache: None 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.wal_filter: None 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:23:00.743 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.two_write_queues: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.wal_compression: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.atomic_flush: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.log_readahead_size: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:23:00.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_background_jobs: 2 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_background_compactions: -1 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_subcompactions: 1 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:23:00.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_open_files: -1 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_background_flushes: -1 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Compression algorithms supported: 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kZSTD supported: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kXpressCompression supported: 0 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:23:00.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kZlibCompression supported: 1 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kSnappyCompression supported: 1 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kLZ4Compression supported: 1 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: kBZip2Compression supported: 0 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi060/store.db/MANIFEST-000015 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.merge_operator: 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_filter: None 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:23:00.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b6682c3aa0) 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: cache_index_and_filter_blocks: 1 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: pin_top_level_index_and_filter: 1 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_type: 0 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: data_block_index_type: 0 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_shortening: 1 2024-04-06T03:23:00.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: checksum: 4 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: no_block_cache: 0 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache: 0x55b6682c7090 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_name: BinnedLRUCache 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_options: 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: capacity : 536870912 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: num_shard_bits : 4 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: strict_capacity_limit : 0 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: high_pri_pool_ratio: 0.000 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_cache_compressed: (nil) 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: persistent_cache: (nil) 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_size: 4096 2024-04-06T03:23:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_size_deviation: 10 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_restart_interval: 16 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: index_block_restart_interval: 1 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: metadata_block_size: 4096 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: partition_filters: 0 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: use_delta_encoding: 1 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: filter_policy: bloomfilter 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: whole_key_filtering: 1 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: verify_compression: 0 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: read_amp_bytes_per_bit: 0 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: format_version: 5 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: enable_index_compression: 1 2024-04-06T03:23:00.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: block_align: 0 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: max_auto_readahead_size: 262144 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: prepopulate_block_cache: 0 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: initial_auto_readahead_size: 8192 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression: NoCompression 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.num_levels: 7 2024-04-06T03:23:00.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:23:00.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:23:00.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:23:00.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:23:00.755 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.table_properties_collectors: 2024-04-06T03:23:00.756 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.inplace_update_support: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.bloom_locality: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.max_successive_merges: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:23:00.757 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.ttl: 2592000 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enable_blob_files: false 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.min_blob_size: 0 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:23:00.758 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi060/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11808, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dee4c115-09e3-4c0f-8a3f-8497cdfe4bd3 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373780557537, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-06T03:23:00.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373780569696, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2693878, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11809, "largest_seqno": 13053, "table_properties": {"data_size": 2687803, "index_size": 3712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12136, "raw_average_key_size": 22, "raw_value_size": 2676271, "raw_average_value_size": 5059, "num_data_blocks": 169, "num_entries": 529, "num_filter_entries": 529, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373780, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dee4c115-09e3-4c0f-8a3f-8497cdfe4bd3", "db_session_id": "JGTW2PY65OQAP7HFESKZ", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373780569828, "job": 1, "event": "recovery_finished"} 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi060/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:23:00.760 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b6683b8000 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: DB pointer 0x55b6683a2000 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** DB Stats ** 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:23:00.761 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: L0 1/0 2.57 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: L6 1/0 6.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Sum 2/0 8.82 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:23:00.762 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 438.5 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Cumulative compaction: 0.00 GB write, 172.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Interval compaction: 0.00 GB write, 172.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:23:00.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Block cache BinnedLRUCache@0x55b6682c7090#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: starting mon.smithi060 rank 0 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi060 fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???) e2 preinit fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).mds e7 new map 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).mds e7 print_map 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: e7 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: legacy client fscid: 1 2024-04-06T03:23:00.764 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Filesystem 'foofs' (1) 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: fs_name foofs 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: epoch 5 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: created 2024-04-06T03:17:21.166439+0000 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: modified 2024-04-06T03:17:27.872964+0000 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: tableserver 0 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: root 0 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: session_timeout 60 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: session_autoclose 300 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: max_file_size 1099511627776 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: max_xattr_size 65536 2024-04-06T03:23:00.765 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: required_client_features {} 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: last_failure 0 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: last_failure_osd_epoch 0 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: max_mds 1 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: in 0 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: up {0=14486} 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: failed 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: damaged 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: stopped 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: data_pools [3] 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: metadata_pool 2 2024-04-06T03:23:00.766 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: inline_data disabled 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: balancer 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: bal_rank_mask -1 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: standby_count_wanted 1 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: qdb_cluster leader: 0 members: 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: [mds.foofs.smithi060.hrodqu{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: Standby daemons: 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout: [mds.foofs.smithi195.dtvkez{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-06T03:23:00.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:00.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:00.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:00.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 ceph-mon[67116]: mon.smithi060@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-04-06T03:23:00.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 bash[67098]: f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 2024-04-06T03:23:00.768 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:00 smithi060 systemd[1]: Started Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:23:01.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:01.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:01.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 680 op/s 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: mon.smithi060 calling monitor election 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: mon.smithi060 is new leader, mons smithi060,smithi195 in quorum (ranks 0,1) 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: monmap epoch 2 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: last_changed 2024-04-06T03:12:17.659725+0000 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: created 2024-04-06T03:09:54.234976+0000 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: min_mon_release 17 (quincy) 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: election_strategy: 1 2024-04-06T03:23:01.509 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: 0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.smithi060 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.smithi195 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: mgrmap e33: smithi060.jttwce(active, since 55s), standbys: smithi195.ultkot 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:01.510 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: from='mgr.14778 ' entity='' 2024-04-06T03:23:01.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:01 smithi060 ceph-mon[67116]: mgrmap e34: smithi060.jttwce(active, since 55s), standbys: smithi195.ultkot 2024-04-06T03:23:01.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 152002 11.56 MB/sec execute 185 sec latency 41.715 ms 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: from='mgr.14778 172.21.15.60:0/3333061009' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 680 op/s 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: mon.smithi060 calling monitor election 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: mon.smithi060 is new leader, mons smithi060,smithi195 in quorum (ranks 0,1) 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: monmap epoch 2 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:01.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: last_changed 2024-04-06T03:12:17.659725+0000 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: created 2024-04-06T03:09:54.234976+0000 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: min_mon_release 17 (quincy) 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: election_strategy: 1 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: 0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.smithi060 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.smithi195 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: mgrmap e33: smithi060.jttwce(active, since 55s), standbys: smithi195.ultkot 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: from='mgr.14778 ' entity='' 2024-04-06T03:23:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:01 smithi195 ceph-mon[27147]: mgrmap e34: smithi060.jttwce(active, since 55s), standbys: smithi195.ultkot 2024-04-06T03:23:02.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 152457 11.59 MB/sec execute 186 sec latency 39.763 ms 2024-04-06T03:23:03.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:03 smithi060 ceph-mon[67116]: mgrmap e35: smithi060.jttwce(active, since 56s), standbys: smithi195.ultkot 2024-04-06T03:23:03.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:03 smithi195 ceph-mon[27147]: mgrmap e35: smithi060.jttwce(active, since 56s), standbys: smithi195.ultkot 2024-04-06T03:23:03.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 153124 11.59 MB/sec execute 187 sec latency 25.343 ms 2024-04-06T03:23:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 153635 11.60 MB/sec execute 188 sec latency 39.563 ms 2024-04-06T03:23:05.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 154114 11.61 MB/sec execute 189 sec latency 65.260 ms 2024-04-06T03:23:06.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 154521 11.61 MB/sec execute 190 sec latency 48.636 ms 2024-04-06T03:23:07.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 154903 11.62 MB/sec execute 191 sec latency 59.627 ms 2024-04-06T03:23:08.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 155325 11.62 MB/sec execute 192 sec latency 33.428 ms 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:23:09.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:09 smithi060 ceph-mon[67116]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:23:09.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 155639 11.61 MB/sec execute 193 sec latency 45.022 ms 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot restarted 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: Standby manager daemon smithi195.ultkot started 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/crt"}]: dispatch 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi195.ultkot/key"}]: dispatch 2024-04-06T03:23:09.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:09 smithi195 ceph-mon[27147]: from='mgr.? 172.21.15.195:0/3749923371' entity='mgr.smithi195.ultkot' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:23:10.589 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:10 smithi060 ceph-mon[67116]: mgrmap e36: smithi060.jttwce(active, since 63s), standbys: smithi195.ultkot 2024-04-06T03:23:10.631 INFO:teuthology.orchestra.run.smithi060.stdout: 5 156183 11.63 MB/sec execute 194 sec latency 31.961 ms 2024-04-06T03:23:10.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:10 smithi195 ceph-mon[27147]: mgrmap e36: smithi060.jttwce(active, since 63s), standbys: smithi195.ultkot 2024-04-06T03:23:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 156700 11.61 MB/sec execute 195 sec latency 42.062 ms 2024-04-06T03:23:12.484 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: Active manager daemon smithi060.jttwce restarted 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: Activating manager daemon smithi060.jttwce 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: mgrmap e37: smithi060.jttwce(active, starting, since 0.00832777s), standbys: smithi195.ultkot 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:23:12.485 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:23:12.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: Active manager daemon smithi060.jttwce restarted 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: Activating manager daemon smithi060.jttwce 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: mgrmap e37: smithi060.jttwce(active, starting, since 0.00832777s), standbys: smithi195.ultkot 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:12.530 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi060.jttwce", "id": "smithi060.jttwce"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr metadata", "who": "smithi195.ultkot", "id": "smithi195.ultkot"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:23:12.531 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:12 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:23:12.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 156999 11.59 MB/sec execute 196 sec latency 49.466 ms 2024-04-06T03:23:13.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:23:13.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:13.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:13.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:23:13.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:23:13.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:23:13.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:13 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:23:13.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 157357 11.58 MB/sec execute 197 sec latency 80.495 ms 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: Manager daemon smithi060.jttwce is now available 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:23:13.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:13 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi060.jttwce/trash_purge_schedule"}]: dispatch 2024-04-06T03:23:14.520 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:14 smithi060 ceph-mon[67116]: mgrmap e38: smithi060.jttwce(active, since 1.07939s), standbys: smithi195.ultkot 2024-04-06T03:23:14.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 157723 11.57 MB/sec execute 198 sec latency 50.984 ms 2024-04-06T03:23:14.680 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:14 smithi195 ceph-mon[27147]: mgrmap e38: smithi060.jttwce(active, since 1.07939s), standbys: smithi195.ultkot 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: [06/Apr/2024:03:23:13] ENGINE Bus STARTING 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: [06/Apr/2024:03:23:13] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: [06/Apr/2024:03:23:13] ENGINE Serving on http://172.21.15.60:8765 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: [06/Apr/2024:03:23:13] ENGINE Bus STARTED 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:15.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:23:15.278 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: Cluster is now healthy 2024-04-06T03:23:15.278 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:15 smithi060 ceph-mon[67116]: mgrmap e39: smithi060.jttwce(active, since 2s), standbys: smithi195.ultkot 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: [06/Apr/2024:03:23:13] ENGINE Bus STARTING 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: [06/Apr/2024:03:23:13] ENGINE Serving on https://172.21.15.60:7150 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: [06/Apr/2024:03:23:13] ENGINE Serving on http://172.21.15.60:8765 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: [06/Apr/2024:03:23:13] ENGINE Bus STARTED 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: pgmap v4: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: Cluster is now healthy 2024-04-06T03:23:15.547 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:15 smithi195 ceph-mon[27147]: mgrmap e39: smithi060.jttwce(active, since 2s), standbys: smithi195.ultkot 2024-04-06T03:23:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 158226 11.61 MB/sec execute 199 sec latency 34.817 ms 2024-04-06T03:23:16.632 INFO:teuthology.orchestra.run.smithi060.stdout: 5 158867 11.63 MB/sec execute 200 sec latency 25.144 ms 2024-04-06T03:23:17.295 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:17 smithi060 ceph-mon[67116]: pgmap v5: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:17.295 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:17 smithi060 ceph-mon[67116]: mgrmap e40: smithi060.jttwce(active, since 4s), standbys: smithi195.ultkot 2024-04-06T03:23:17.295 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:17 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:17.295 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:17 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:17.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 159353 11.65 MB/sec execute 201 sec latency 36.213 ms 2024-04-06T03:23:17.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:17 smithi195 ceph-mon[27147]: pgmap v5: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:17.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:17 smithi195 ceph-mon[27147]: mgrmap e40: smithi060.jttwce(active, since 4s), standbys: smithi195.ultkot 2024-04-06T03:23:17.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:17 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:17.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:17 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:18.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 159946 11.66 MB/sec execute 202 sec latency 33.527 ms 2024-04-06T03:23:19.307 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:19 smithi060 ceph-mon[67116]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:19.307 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:19 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:19.307 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:19 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:19.320 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:19 smithi195 ceph-mon[27147]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-04-06T03:23:19.320 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:19 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:19.320 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:19 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:19.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 160434 11.65 MB/sec execute 203 sec latency 44.886 ms 2024-04-06T03:23:20.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 160950 11.65 MB/sec execute 204 sec latency 30.656 ms 2024-04-06T03:23:21.326 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:21 smithi060 ceph-mon[67116]: pgmap v7: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 5.8 MiB/s wr, 532 op/s 2024-04-06T03:23:21.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 161439 11.67 MB/sec execute 205 sec latency 43.873 ms 2024-04-06T03:23:21.662 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:21 smithi195 ceph-mon[27147]: pgmap v7: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 5.8 MiB/s wr, 532 op/s 2024-04-06T03:23:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 161811 11.69 MB/sec execute 206 sec latency 46.913 ms 2024-04-06T03:23:23.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:23 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:23 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:23 smithi060 ceph-mon[67116]: pgmap v8: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.5 MiB/s wr, 413 op/s 2024-04-06T03:23:23.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:23 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.116 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:23 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:23 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:23 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:23 smithi195 ceph-mon[27147]: pgmap v8: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.5 MiB/s wr, 413 op/s 2024-04-06T03:23:23.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:23 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:23 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:23.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 162164 11.67 MB/sec execute 207 sec latency 40.703 ms 2024-04-06T03:23:24.554 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:23:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 162467 11.65 MB/sec execute 208 sec latency 55.873 ms 2024-04-06T03:23:25.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:25 smithi060 ceph-mon[67116]: pgmap v9: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-04-06T03:23:25.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:25 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:25.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:25 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:25.414 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:25.414 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:25.458 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (2m) 2s ago 11m 19.9M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (11m) 2s ago 11m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (11m) 8s ago 11m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 unknown 2s ago 11m - - 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (6m) 2s ago 5m 40.4M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (5m) 8s ago 5m 15.2M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (98s) 2s ago 13m 478M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (45s) 8s ago 11m 447M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (25s) 2s ago 13m 29.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (11m) 8s ago 11m 52.0M 2048M 17.2.0 e1d6a67b021e 87271f316077 2024-04-06T03:23:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (5m) 2s ago 5m 271M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (5m) 8s ago 5m 81.0M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (2m) 2s ago 11m 9.98M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (110s) 8s ago 10m 18.7M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (10m) 2s ago 10m 570M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (9m) 2s ago 9m 488M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (9m) 2s ago 9m 357M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (9m) 2s ago 9m 492M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (8m) 8s ago 8m 512M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (8m) 8s ago 8m 325M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (7m) 8s ago 7m 509M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (7m) 8s ago 7m 431M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:23:25.460 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (52s) 2s ago 10m 53.6M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:23:25.633 INFO:teuthology.orchestra.run.smithi060.stdout: 5 162704 11.62 MB/sec execute 209 sec latency 41.808 ms 2024-04-06T03:23:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:25 smithi195 ceph-mon[27147]: pgmap v9: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-04-06T03:23:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:25 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:25 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:25 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:25 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:23:25.984 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:23:25.985 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:23:26.402 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr" 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "3/23 daemons upgraded", 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:23:26.403 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:23:26.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:23:26.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:23:26.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:26.437 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:23:26.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T03:23:26.439 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:26 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2024224724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:23:26.607 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-06T03:23:26.607 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:26.608 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:26.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:23:26.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T03:23:26.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:26 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/2024224724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:23:26.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 162902 11.60 MB/sec execute 210 sec latency 68.161 ms 2024-04-06T03:23:26.910 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:26.910 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: pgmap v10: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 6.4 MiB/s wr, 616 op/s 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:23:27.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:27 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3400138162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:23:27.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 163153 11.58 MB/sec execute 211 sec latency 104.227 ms 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: Updating smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: Updating smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: pgmap v10: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 6.4 MiB/s wr, 616 op/s 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:23:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:27 smithi195 ceph-mon[27147]: from='client.? 172.21.15.60:0/3400138162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:23:28.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:28 smithi195 ceph-mon[27147]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:28.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:28.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:28.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='mgr.24613 ' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:28.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:23:28.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:23:28.497 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:28.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 163587 11.57 MB/sec execute 212 sec latency 59.664 ms 2024-04-06T03:23:29.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:29 smithi195 ceph-mon[27147]: pgmap v11: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 6.4 MiB/s wr, 616 op/s 2024-04-06T03:23:29.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:29 smithi195 ceph-mon[27147]: Upgrade: Updating mon.smithi195 2024-04-06T03:23:29.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:29 smithi195 ceph-mon[27147]: Deploying daemon mon.smithi195 on smithi195 2024-04-06T03:23:29.503 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:29 smithi060 ceph-mon[67116]: pgmap v11: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 6.4 MiB/s wr, 616 op/s 2024-04-06T03:23:29.503 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:29 smithi060 ceph-mon[67116]: Upgrade: Updating mon.smithi195 2024-04-06T03:23:29.503 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:29 smithi060 ceph-mon[67116]: Deploying daemon mon.smithi195 on smithi195 2024-04-06T03:23:29.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 164193 11.57 MB/sec execute 213 sec latency 43.937 ms 2024-04-06T03:23:30.577 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:30 smithi195 systemd[1]: Stopping Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:23:30.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 164945 11.62 MB/sec execute 214 sec latency 32.709 ms 2024-04-06T03:23:30.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195[27127]: 2024-04-06T03:23:30.577+0000 7ff3e1d0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi195 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:23:30.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195[27127]: 2024-04-06T03:23:30.577+0000 7ff3e1d0d700 -1 mon.smithi195@1(peon) e2 *** Got Signal Terminated *** 2024-04-06T03:23:30.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:30 smithi195 podman[53002]: 2024-04-06 03:23:30.639435639 +0000 UTC m=+0.189895238 container died 87271f316077731370ef19df27b101b599613367370a695c6b157f170bfbccf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, release=754, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T03:23:31.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 165712 11.68 MB/sec execute 215 sec latency 34.239 ms 2024-04-06T03:23:31.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:31 smithi195 podman[53002]: 2024-04-06 03:23:31.470211475 +0000 UTC m=+1.020671067 container cleanup 87271f316077731370ef19df27b101b599613367370a695c6b157f170bfbccf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, GIT_BRANCH=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container) 2024-04-06T03:23:31.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:31 smithi195 bash[53002]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195 2024-04-06T03:23:32.082 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:31 smithi195 podman[53014]: 2024-04-06 03:23:31.78872263 +0000 UTC m=+1.149833962 container remove 87271f316077731370ef19df27b101b599613367370a695c6b157f170bfbccf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.openshift.expose-services=, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, name=centos-stream) 2024-04-06T03:23:32.355 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service: Deactivated successfully. 2024-04-06T03:23:32.356 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 systemd[1]: Stopped Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:23:32.356 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service: Consumed 7.268s CPU time. 2024-04-06T03:23:32.629 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 systemd[1]: Starting Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:23:32.629 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 podman[53139]: 2024-04-06 03:23:32.506116843 +0000 UTC m=+0.022772077 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:23:32.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 166274 11.69 MB/sec execute 216 sec latency 48.927 ms 2024-04-06T03:23:32.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:32 smithi195 podman[53139]: 2024-04-06 03:23:32.632288737 +0000 UTC m=+0.148943945 container create 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T03:23:33.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 166793 11.70 MB/sec execute 217 sec latency 70.450 ms 2024-04-06T03:23:34.218 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 podman[53139]: 2024-04-06 03:23:33.803081255 +0000 UTC m=+1.319736473 container init 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 podman[53139]: 2024-04-06 03:23:33.807784979 +0000 UTC m=+1.324440185 container start 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 2 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: pidfile_write: ignore empty --pid-file 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: load: jerasure load: lrc 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: RocksDB version: 7.9.2 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Git sha 0 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: DB SUMMARY 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: DB Session ID: 04QPE8EMSVKJHMY66B9L 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: CURRENT file: CURRENT 2024-04-06T03:23:34.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi195/store.db dir, Total Num: 1, files: 000024.sst 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi195/store.db: 000022.log size: 7616032 ; 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.error_if_exists: 0 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.create_if_missing: 0 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.paranoid_checks: 1 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.env: 0x5638e9697c60 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.info_log: 0x5638e9fff9a0 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:23:34.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.statistics: (nil) 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.use_fsync: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_log_file_size: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_fallocate: 1 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.use_direct_reads: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.db_log_dir: 2024-04-06T03:23:34.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.wal_dir: 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.write_buffer_manager: 0x5638e9fd21e0 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:23:34.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.unordered_write: 0 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.row_cache: None 2024-04-06T03:23:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.wal_filter: None 2024-04-06T03:23:34.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:23:34.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:23:34.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.two_write_queues: 0 2024-04-06T03:23:34.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.wal_compression: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.atomic_flush: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.log_readahead_size: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:23:34.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_background_jobs: 2 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_background_compactions: -1 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_subcompactions: 1 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_open_files: -1 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:23:34.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_background_flushes: -1 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Compression algorithms supported: 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kZSTD supported: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kXpressCompression supported: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kZlibCompression supported: 1 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kSnappyCompression supported: 1 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kLZ4Compression supported: 1 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: kBZip2Compression supported: 0 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:23:34.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi195/store.db/MANIFEST-000009 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.merge_operator: 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_filter: None 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5638e9fffaa0) 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: cache_index_and_filter_blocks: 1 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: pin_top_level_index_and_filter: 1 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: index_type: 0 2024-04-06T03:23:34.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: data_block_index_type: 0 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: index_shortening: 1 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: checksum: 4 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: no_block_cache: 0 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_cache: 0x5638ea003090 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_cache_name: BinnedLRUCache 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_cache_options: 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: capacity : 536870912 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: num_shard_bits : 4 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: strict_capacity_limit : 0 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: high_pri_pool_ratio: 0.000 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_cache_compressed: (nil) 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: persistent_cache: (nil) 2024-04-06T03:23:34.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_size: 4096 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_size_deviation: 10 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_restart_interval: 16 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: index_block_restart_interval: 1 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: metadata_block_size: 4096 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: partition_filters: 0 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: use_delta_encoding: 1 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: filter_policy: bloomfilter 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: whole_key_filtering: 1 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: verify_compression: 0 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: read_amp_bytes_per_bit: 0 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: format_version: 5 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: enable_index_compression: 1 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: block_align: 0 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: max_auto_readahead_size: 262144 2024-04-06T03:23:34.230 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: prepopulate_block_cache: 0 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: initial_auto_readahead_size: 8192 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression: NoCompression 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.num_levels: 7 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:23:34.231 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:23:34.232 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:23:34.233 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:23:34.234 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.table_properties_collectors: 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.inplace_update_support: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.bloom_locality: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.max_successive_merges: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:23:34.235 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.ttl: 2592000 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enable_blob_files: false 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.min_blob_size: 0 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:23:34.236 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi195/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12186, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 87f47545-cbab-443d-be39-9693030af09f 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373813840356, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373813861378, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4635889, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12191, "largest_seqno": 14120, "table_properties": {"data_size": 4627525, "index_size": 5424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17450, "raw_average_key_size": 23, "raw_value_size": 4611094, "raw_average_value_size": 6083, "num_data_blocks": 246, "num_entries": 758, "num_filter_entries": 758, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373813, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "87f47545-cbab-443d-be39-9693030af09f", "db_session_id": "04QPE8EMSVKJHMY66B9L", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712373813861496, "job": 1, "event": "recovery_finished"} 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-04-06T03:23:34.237 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi195/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5638ea0f4000 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: DB pointer 0x5638ea0de000 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: ** DB Stats ** 2024-04-06T03:23:34.238 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: ** Compaction Stats [default] ** 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:23:34.240 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: L0 1/0 4.42 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: L6 1/0 6.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Sum 2/0 10.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: ** Compaction Stats [default] ** 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:23:34.241 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 526.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Cumulative compaction: 0.00 GB write, 185.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:23:34.242 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Interval compaction: 0.00 GB write, 185.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Block cache BinnedLRUCache@0x5638ea003090#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: starting mon.smithi195 rank 1 at public addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] at bind addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi195 fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???) e2 preinit fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).mds e7 new map 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).mds e7 print_map 2024-04-06T03:23:34.243 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: e7 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: legacy client fscid: 1 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Filesystem 'foofs' (1) 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: fs_name foofs 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: epoch 5 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: created 2024-04-06T03:17:21.166439+0000 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: modified 2024-04-06T03:17:27.872964+0000 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: tableserver 0 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: root 0 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: session_timeout 60 2024-04-06T03:23:34.244 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: session_autoclose 300 2024-04-06T03:23:34.245 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: max_file_size 1099511627776 2024-04-06T03:23:34.245 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: max_xattr_size 65536 2024-04-06T03:23:34.245 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: required_client_features {} 2024-04-06T03:23:34.245 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: last_failure 0 2024-04-06T03:23:34.245 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: last_failure_osd_epoch 0 2024-04-06T03:23:34.246 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T03:23:34.246 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: max_mds 1 2024-04-06T03:23:34.246 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: in 0 2024-04-06T03:23:34.246 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: up {0=14486} 2024-04-06T03:23:34.246 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: failed 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: damaged 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: stopped 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: data_pools [3] 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: metadata_pool 2 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: inline_data disabled 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: balancer 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: bal_rank_mask -1 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: standby_count_wanted 1 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: qdb_cluster leader: 0 members: 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: [mds.foofs.smithi060.hrodqu{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.60:6834/970676067,v1:172.21.15.60:6835/970676067] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T03:23:34.247 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: Standby daemons: 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout: [mds.foofs.smithi195.dtvkez{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 ceph-mon[53157]: mon.smithi195@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 bash[53139]: 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:33 smithi195 systemd[1]: Started Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:34 smithi195 ceph-mon[53157]: pgmap v12: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 792 op/s 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:34 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:34.248 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:34 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:34.290 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:34 smithi060 ceph-mon[67116]: pgmap v12: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 792 op/s 2024-04-06T03:23:34.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:34 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:34.291 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:34 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:34.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 167226 11.69 MB/sec execute 218 sec latency 55.808 ms 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: pgmap v14: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 871 op/s 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: mgrmap e40: smithi060.jttwce(active, since 22s), standbys: smithi195.ultkot 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: mon.smithi060 calling monitor election 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: mon.smithi195 calling monitor election 2024-04-06T03:23:35.317 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: mon.smithi060 is new leader, mons smithi060,smithi195 in quorum (ranks 0,1) 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: monmap epoch 3 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: last_changed 2024-04-06T03:23:34.223274+0000 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: created 2024-04-06T03:09:54.234976+0000 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: min_mon_release 19 (squid) 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: election_strategy: 1 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: 0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.smithi060 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.smithi195 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: mgrmap e40: smithi060.jttwce(active, since 22s), standbys: smithi195.ultkot 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:35.318 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:35.319 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:35.319 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:35.319 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:35.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 167551 11.68 MB/sec execute 219 sec latency 46.165 ms 2024-04-06T03:23:35.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: pgmap v14: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 871 op/s 2024-04-06T03:23:35.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:35.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: mgrmap e40: smithi060.jttwce(active, since 22s), standbys: smithi195.ultkot 2024-04-06T03:23:35.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi060"}]: dispatch 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mon metadata", "id": "smithi195"}]: dispatch 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: mon.smithi060 calling monitor election 2024-04-06T03:23:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: mon.smithi195 calling monitor election 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: mon.smithi060 is new leader, mons smithi060,smithi195 in quorum (ranks 0,1) 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: monmap epoch 3 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: last_changed 2024-04-06T03:23:34.223274+0000 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: created 2024-04-06T03:09:54.234976+0000 2024-04-06T03:23:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: min_mon_release 19 (squid) 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: election_strategy: 1 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: 0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.smithi060 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.smithi195 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: mgrmap e40: smithi060.jttwce(active, since 22s), standbys: smithi195.ultkot 2024-04-06T03:23:35.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:23:35.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:23:35.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:23:35.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:35.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:35.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:36.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 167830 11.65 MB/sec execute 220 sec latency 70.638 ms 2024-04-06T03:23:37.432 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:37 smithi060 ceph-mon[67116]: pgmap v15: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-06T03:23:37.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 168244 11.65 MB/sec execute 221 sec latency 56.631 ms 2024-04-06T03:23:37.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:37 smithi195 ceph-mon[53157]: pgmap v15: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-06T03:23:38.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 168598 11.65 MB/sec execute 222 sec latency 31.565 ms 2024-04-06T03:23:39.466 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:39 smithi060 ceph-mon[67116]: pgmap v16: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.7 MiB/s wr, 551 op/s 2024-04-06T03:23:39.467 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:39.467 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:39.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 168934 11.65 MB/sec execute 223 sec latency 64.611 ms 2024-04-06T03:23:39.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:39 smithi195 ceph-mon[53157]: pgmap v16: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.7 MiB/s wr, 551 op/s 2024-04-06T03:23:39.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:39.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:40.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 169241 11.64 MB/sec execute 224 sec latency 74.152 ms 2024-04-06T03:23:41.261 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:41 smithi060 ceph-mon[67116]: pgmap v17: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 780 op/s 2024-04-06T03:23:41.261 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:41 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:41.261 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:41 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:41.324 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:41 smithi195 ceph-mon[53157]: pgmap v17: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 780 op/s 2024-04-06T03:23:41.324 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:41 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:41.324 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:41 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 169556 11.62 MB/sec execute 225 sec latency 67.372 ms 2024-04-06T03:23:42.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 169779 11.59 MB/sec execute 226 sec latency 52.894 ms 2024-04-06T03:23:43.287 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:43 smithi060 ceph-mon[67116]: pgmap v18: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 611 op/s 2024-04-06T03:23:43.287 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 170009 11.57 MB/sec execute 227 sec latency 70.112 ms 2024-04-06T03:23:43.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:43 smithi195 ceph-mon[53157]: pgmap v18: 97 pgs: 97 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 611 op/s 2024-04-06T03:23:43.652 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 5 170270 11.55 MB/sec execute 228 sec latency 51.346 ms 2024-04-06T03:23:45.082 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: pgmap v19: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 791 op/s 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: pgmap v20: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 475 op/s 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: pgmap v21: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: Reconfiguring mon.smithi060 (monmap changed)... 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:23:45.083 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:45.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:45 smithi060 ceph-mon[67116]: Reconfiguring daemon mon.smithi060 on smithi060 2024-04-06T03:23:45.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: pgmap v19: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 791 op/s 2024-04-06T03:23:45.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: pgmap v20: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 475 op/s 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: pgmap v21: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: Reconfiguring mon.smithi060 (monmap changed)... 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:45.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:45 smithi195 ceph-mon[53157]: Reconfiguring daemon mon.smithi060 on smithi060 2024-04-06T03:23:45.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 170603 11.54 MB/sec execute 229 sec latency 58.132 ms 2024-04-06T03:23:46.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 170905 11.51 MB/sec execute 230 sec latency 70.732 ms 2024-04-06T03:23:46.680 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:46 smithi060 ceph-mon[67116]: pgmap v22: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-04-06T03:23:46.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:46 smithi195 ceph-mon[53157]: pgmap v22: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-04-06T03:23:47.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 171149 11.49 MB/sec execute 231 sec latency 49.043 ms 2024-04-06T03:23:47.945 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:47.945 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:47.945 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: Reconfiguring mgr.smithi060.jttwce (monmap changed)... 2024-04-06T03:23:47.946 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:23:47.946 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:23:47.946 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:47.946 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:47 smithi060 ceph-mon[67116]: Reconfiguring daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:23:48.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:48.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:48.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: Reconfiguring mgr.smithi060.jttwce (monmap changed)... 2024-04-06T03:23:48.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi060.jttwce", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:23:48.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:23:48.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:48.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:47 smithi195 ceph-mon[53157]: Reconfiguring daemon mgr.smithi060.jttwce on smithi060 2024-04-06T03:23:48.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 171485 11.47 MB/sec execute 232 sec latency 49.887 ms 2024-04-06T03:23:49.006 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:48 smithi060 ceph-mon[67116]: pgmap v23: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 259 op/s 2024-04-06T03:23:49.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:48 smithi195 ceph-mon[53157]: pgmap v23: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 259 op/s 2024-04-06T03:23:49.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 171763 11.46 MB/sec execute 233 sec latency 67.247 ms 2024-04-06T03:23:50.557 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: Reconfiguring crash.smithi060 (monmap changed)... 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: Reconfiguring daemon crash.smithi060 on smithi060 2024-04-06T03:23:50.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:50 smithi060 ceph-mon[67116]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 526 op/s 2024-04-06T03:23:50.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 172044 11.43 MB/sec execute 234 sec latency 47.458 ms 2024-04-06T03:23:50.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:50.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:50.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: Reconfiguring crash.smithi060 (monmap changed)... 2024-04-06T03:23:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:23:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: Reconfiguring daemon crash.smithi060 on smithi060 2024-04-06T03:23:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:50 smithi195 ceph-mon[53157]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 526 op/s 2024-04-06T03:23:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 172342 11.44 MB/sec execute 235 sec latency 41.770 ms 2024-04-06T03:23:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 172570 11.43 MB/sec execute 236 sec latency 71.168 ms 2024-04-06T03:23:53.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:53.104 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:53.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T03:23:53.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:23:53.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:53.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.0 on smithi060 2024-04-06T03:23:53.105 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:52 smithi060 ceph-mon[67116]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-04-06T03:23:53.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:53.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T03:23:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:23:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.0 on smithi060 2024-04-06T03:23:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:52 smithi195 ceph-mon[53157]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-04-06T03:23:53.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 173035 11.43 MB/sec execute 237 sec latency 40.802 ms 2024-04-06T03:23:54.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 173349 11.41 MB/sec execute 238 sec latency 47.029 ms 2024-04-06T03:23:55.134 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:55 smithi060 ceph-mon[67116]: pgmap v26: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 434 op/s 2024-04-06T03:23:55.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:55 smithi195 ceph-mon[53157]: pgmap v26: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 434 op/s 2024-04-06T03:23:55.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 173672 11.40 MB/sec execute 239 sec latency 45.195 ms 2024-04-06T03:23:56.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 173967 11.39 MB/sec execute 240 sec latency 82.539 ms 2024-04-06T03:23:57.198 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.1 on smithi060 2024-04-06T03:23:57.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:56 smithi060 ceph-mon[67116]: pgmap v27: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.1 on smithi060 2024-04-06T03:23:57.219 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:56 smithi195 ceph-mon[53157]: pgmap v27: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-04-06T03:23:57.467 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:23:57.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 174255 11.37 MB/sec execute 241 sec latency 61.648 ms 2024-04-06T03:23:57.996 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:58.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (3m) 35s ago 12m 19.9M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (12m) 35s ago 12m 6409k - 17.2.0 e1d6a67b021e 1530e56e69e2 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (11m) 19s ago 11m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 unknown 35s ago 12m - - 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (6m) 35s ago 6m 40.4M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (6m) 19s ago 6m 15.4M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:23:58.506 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (2m) 35s ago 13m 478M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (78s) 19s ago 11m 448M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (58s) 35s ago 14m 29.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (25s) 19s ago 11m 27.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (6m) 35s ago 6m 271M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (6m) 19s ago 6m 81.1M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (2m) 35s ago 12m 9.98M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (2m) 19s ago 11m 19.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (11m) 35s ago 11m 570M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:23:58.507 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (10m) 35s ago 10m 488M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (10m) 35s ago 10m 357M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (9m) 35s ago 9m 492M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (9m) 19s ago 9m 532M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (8m) 19s ago 8m 363M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (8m) 19s ago 8m 547M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (8m) 19s ago 8m 481M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:23:58.508 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (85s) 35s ago 11m 53.6M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:23:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 174534 11.34 MB/sec execute 242 sec latency 43.358 ms 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: pgmap v28: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:23:59.024 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:23:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:23:59.154 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:23:59.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:59.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:23:59.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: pgmap v28: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 362 op/s 2024-04-06T03:23:59.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:59.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:23:59.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:23:59.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 174738 11.32 MB/sec execute 243 sec latency 65.804 ms 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "mon" 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "4/23 daemons upgraded", 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:23:59.695 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:24:00.064 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:59 smithi060 ceph-mon[67116]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:00.064 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:59 smithi060 ceph-mon[67116]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T03:24:00.064 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:59 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.2 on smithi060 2024-04-06T03:24:00.064 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:23:59 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1719581064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:00.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:59 smithi195 ceph-mon[53157]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:00.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:59 smithi195 ceph-mon[53157]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T03:24:00.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:59 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.2 on smithi060 2024-04-06T03:24:00.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:23:59 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1719581064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:00.321 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:24:00.322 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:24:00.322 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:24:00.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 175108 11.31 MB/sec execute 244 sec latency 56.946 ms 2024-04-06T03:24:01.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:00 smithi060 ceph-mon[67116]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:01.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:00 smithi060 ceph-mon[67116]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 567 op/s 2024-04-06T03:24:01.084 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:00 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3592624835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:24:01.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:00 smithi195 ceph-mon[53157]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:01.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:00 smithi195 ceph-mon[53157]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.5 MiB/s wr, 567 op/s 2024-04-06T03:24:01.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:00 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3592624835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:24:01.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 175442 11.30 MB/sec execute 245 sec latency 50.490 ms 2024-04-06T03:24:02.362 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:02.362 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:02.363 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T03:24:02.363 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:24:02.363 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:02.363 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:02 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.3 on smithi060 2024-04-06T03:24:02.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:02.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:02.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T03:24:02.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:24:02.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:02.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:02 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.3 on smithi060 2024-04-06T03:24:02.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 175779 11.30 MB/sec execute 246 sec latency 37.335 ms 2024-04-06T03:24:03.369 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:03 smithi060 ceph-mon[67116]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-04-06T03:24:03.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:03 smithi195 ceph-mon[53157]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-04-06T03:24:03.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 176177 11.32 MB/sec execute 247 sec latency 53.669 ms 2024-04-06T03:24:04.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 176634 11.31 MB/sec execute 248 sec latency 28.090 ms 2024-04-06T03:24:05.030 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:05.031 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: pgmap v31: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: Reconfiguring mds.foofs.smithi060.hrodqu (monmap changed)... 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:05.123 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:05 smithi060 ceph-mon[67116]: Reconfiguring daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:24:05.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: pgmap v31: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-04-06T03:24:05.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: Reconfiguring mds.foofs.smithi060.hrodqu (monmap changed)... 2024-04-06T03:24:05.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:24:05.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:05.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:05 smithi195 ceph-mon[53157]: Reconfiguring daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:24:05.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 177067 11.32 MB/sec execute 249 sec latency 31.648 ms 2024-04-06T03:24:06.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 177611 11.32 MB/sec execute 250 sec latency 63.164 ms 2024-04-06T03:24:07.154 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:07 smithi060 ceph-mon[67116]: pgmap v32: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-06T03:24:07.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:07.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:07.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:07.155 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:07.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:07 smithi195 ceph-mon[53157]: pgmap v32: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-06T03:24:07.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:07.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:07.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:07.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:07.635 INFO:teuthology.orchestra.run.smithi060.stdout: 5 178081 11.31 MB/sec execute 251 sec latency 60.921 ms 2024-04-06T03:24:08.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:08 smithi060 ceph-mon[67116]: Reconfiguring crash.smithi195 (monmap changed)... 2024-04-06T03:24:08.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:08 smithi060 ceph-mon[67116]: Reconfiguring daemon crash.smithi195 on smithi195 2024-04-06T03:24:08.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:08 smithi195 ceph-mon[53157]: Reconfiguring crash.smithi195 (monmap changed)... 2024-04-06T03:24:08.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:08 smithi195 ceph-mon[53157]: Reconfiguring daemon crash.smithi195 on smithi195 2024-04-06T03:24:08.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 178581 11.33 MB/sec execute 252 sec latency 48.573 ms 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: pgmap v33: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:24:09.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:09 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:09.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: pgmap v33: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-06T03:24:09.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:09.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:09.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi195.ultkot", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:24:09.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:24:09.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:09 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:09.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 179125 11.33 MB/sec execute 253 sec latency 31.771 ms 2024-04-06T03:24:10.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:10 smithi195 ceph-mon[53157]: Reconfiguring mgr.smithi195.ultkot (monmap changed)... 2024-04-06T03:24:10.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:10 smithi195 ceph-mon[53157]: Reconfiguring daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:24:10.309 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:10 smithi060 ceph-mon[67116]: Reconfiguring mgr.smithi195.ultkot (monmap changed)... 2024-04-06T03:24:10.310 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:10 smithi060 ceph-mon[67116]: Reconfiguring daemon mgr.smithi195.ultkot on smithi195 2024-04-06T03:24:10.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 179586 11.36 MB/sec execute 254 sec latency 35.980 ms 2024-04-06T03:24:11.314 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: pgmap v34: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.7 MiB/s wr, 688 op/s 2024-04-06T03:24:11.315 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:11.315 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:11.315 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:24:11.315 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:24:11.315 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:11.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: pgmap v34: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.7 MiB/s wr, 688 op/s 2024-04-06T03:24:11.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:11.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:11.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:24:11.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:24:11.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 180113 11.37 MB/sec execute 255 sec latency 38.787 ms 2024-04-06T03:24:12.327 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:12 smithi060 ceph-mon[67116]: Reconfiguring mon.smithi195 (monmap changed)... 2024-04-06T03:24:12.327 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:12 smithi060 ceph-mon[67116]: Reconfiguring daemon mon.smithi195 on smithi195 2024-04-06T03:24:12.363 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:12 smithi195 ceph-mon[53157]: Reconfiguring mon.smithi195 (monmap changed)... 2024-04-06T03:24:12.363 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:12 smithi195 ceph-mon[53157]: Reconfiguring daemon mon.smithi195 on smithi195 2024-04-06T03:24:12.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 180451 11.36 MB/sec execute 256 sec latency 51.571 ms 2024-04-06T03:24:13.333 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: pgmap v35: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s 2024-04-06T03:24:13.333 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:13.333 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:13.334 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:13.334 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:24:13.334 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:13.336 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: pgmap v35: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s 2024-04-06T03:24:13.336 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:13.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:13.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:13.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:24:13.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:13.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 180835 11.36 MB/sec execute 257 sec latency 29.113 ms 2024-04-06T03:24:14.344 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:14 smithi060 ceph-mon[67116]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T03:24:14.344 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:14 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.4 on smithi195 2024-04-06T03:24:14.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:14 smithi195 ceph-mon[53157]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T03:24:14.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:14 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.4 on smithi195 2024-04-06T03:24:14.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 181277 11.35 MB/sec execute 258 sec latency 44.534 ms 2024-04-06T03:24:15.300 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:15 smithi060 ceph-mon[67116]: pgmap v36: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 762 op/s 2024-04-06T03:24:15.300 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:15.300 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:15.300 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:24:15.300 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:15.410 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:15 smithi195 ceph-mon[53157]: pgmap v36: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 762 op/s 2024-04-06T03:24:15.411 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:15.411 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:15.411 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:24:15.411 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:15.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 181667 11.34 MB/sec execute 259 sec latency 46.323 ms 2024-04-06T03:24:16.385 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:16 smithi060 ceph-mon[67116]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T03:24:16.385 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:16 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.5 on smithi195 2024-04-06T03:24:16.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:16 smithi195 ceph-mon[53157]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T03:24:16.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:16 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.5 on smithi195 2024-04-06T03:24:16.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 182028 11.34 MB/sec execute 260 sec latency 46.657 ms 2024-04-06T03:24:17.312 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:17 smithi195 ceph-mon[53157]: pgmap v37: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 576 op/s 2024-04-06T03:24:17.312 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:17.312 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:17.312 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:24:17.312 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:17.404 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:17 smithi060 ceph-mon[67116]: pgmap v37: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 576 op/s 2024-04-06T03:24:17.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:17.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:17.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:24:17.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:17.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 182581 11.34 MB/sec execute 261 sec latency 59.182 ms 2024-04-06T03:24:18.411 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:18 smithi060 ceph-mon[67116]: Reconfiguring osd.6 (monmap changed)... 2024-04-06T03:24:18.411 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:18 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.6 on smithi195 2024-04-06T03:24:18.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:18 smithi195 ceph-mon[53157]: Reconfiguring osd.6 (monmap changed)... 2024-04-06T03:24:18.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:18 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.6 on smithi195 2024-04-06T03:24:18.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 183098 11.38 MB/sec execute 262 sec latency 34.290 ms 2024-04-06T03:24:19.441 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: pgmap v38: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 576 op/s 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: Reconfiguring osd.7 (monmap changed)... 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:19.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:19 smithi060 ceph-mon[67116]: Reconfiguring daemon osd.7 on smithi195 2024-04-06T03:24:19.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: pgmap v38: 97 pgs: 97 active+clean; 355 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 576 op/s 2024-04-06T03:24:19.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:19.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:19.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: Reconfiguring osd.7 (monmap changed)... 2024-04-06T03:24:19.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:24:19.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:19.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:19 smithi195 ceph-mon[53157]: Reconfiguring daemon osd.7 on smithi195 2024-04-06T03:24:19.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 183623 11.39 MB/sec execute 263 sec latency 54.253 ms 2024-04-06T03:24:20.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 183970 11.37 MB/sec execute 264 sec latency 39.843 ms 2024-04-06T03:24:21.219 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:21 smithi060 ceph-mon[67116]: pgmap v39: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 847 op/s 2024-04-06T03:24:21.220 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:21.220 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:21.220 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:24:21.220 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:21.306 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:21 smithi195 ceph-mon[53157]: pgmap v39: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 847 op/s 2024-04-06T03:24:21.306 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:21.306 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:21.306 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:24:21.306 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:21.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 184178 11.36 MB/sec execute 265 sec latency 109.725 ms 2024-04-06T03:24:22.227 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:22 smithi060 ceph-mon[67116]: Reconfiguring mds.foofs.smithi195.dtvkez (monmap changed)... 2024-04-06T03:24:22.228 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:22 smithi060 ceph-mon[67116]: Reconfiguring daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:24:22.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:22 smithi195 ceph-mon[53157]: Reconfiguring mds.foofs.smithi195.dtvkez (monmap changed)... 2024-04-06T03:24:22.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:22 smithi195 ceph-mon[53157]: Reconfiguring daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:24:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 184429 11.34 MB/sec execute 266 sec latency 105.124 ms 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: pgmap v40: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi060"}]: dispatch 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi060"}]': finished 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi195"}]: dispatch 2024-04-06T03:24:23.280 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi195"}]': finished 2024-04-06T03:24:23.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 184742 11.32 MB/sec execute 267 sec latency 63.556 ms 2024-04-06T03:24:23.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: pgmap v40: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.6 MiB/s wr, 550 op/s 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi060"}]: dispatch 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi060"}]': finished 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi195"}]: dispatch 2024-04-06T03:24:23.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi195"}]': finished 2024-04-06T03:24:24.298 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:24 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all mon 2024-04-06T03:24:24.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 185374 11.33 MB/sec execute 268 sec latency 38.502 ms 2024-04-06T03:24:24.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:24 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all mon 2024-04-06T03:24:25.303 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:25 smithi060 ceph-mon[67116]: pgmap v41: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 765 op/s 2024-04-06T03:24:25.304 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:25.304 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:25.304 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:25.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 186089 11.35 MB/sec execute 269 sec latency 40.798 ms 2024-04-06T03:24:25.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:25 smithi195 ceph-mon[53157]: pgmap v41: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 765 op/s 2024-04-06T03:24:25.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:25.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi060", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:25.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:26.332 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:26 smithi060 ceph-mon[67116]: Upgrade: Updating crash.smithi060 (1/2) 2024-04-06T03:24:26.333 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:26 smithi060 ceph-mon[67116]: Deploying daemon crash.smithi060 on smithi060 2024-04-06T03:24:26.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:26 smithi195 ceph-mon[53157]: Upgrade: Updating crash.smithi060 (1/2) 2024-04-06T03:24:26.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:26 smithi195 ceph-mon[53157]: Deploying daemon crash.smithi060 on smithi060 2024-04-06T03:24:26.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 186399 11.36 MB/sec execute 270 sec latency 49.616 ms 2024-04-06T03:24:27.376 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:27 smithi060 ceph-mon[67116]: pgmap v42: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-04-06T03:24:27.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 186728 11.37 MB/sec execute 271 sec latency 65.015 ms 2024-04-06T03:24:27.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:27 smithi195 ceph-mon[53157]: pgmap v42: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-04-06T03:24:28.388 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:28.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 187044 11.34 MB/sec execute 272 sec latency 55.346 ms 2024-04-06T03:24:28.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:29.391 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:29 smithi060 ceph-mon[67116]: pgmap v43: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-04-06T03:24:29.636 INFO:teuthology.orchestra.run.smithi060.stdout: 5 187568 11.36 MB/sec execute 273 sec latency 37.183 ms 2024-04-06T03:24:29.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:29 smithi195 ceph-mon[53157]: pgmap v43: 97 pgs: 97 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-04-06T03:24:30.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 188174 11.37 MB/sec execute 274 sec latency 35.099 ms 2024-04-06T03:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:24:31.403 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:31 smithi060 ceph-mon[67116]: pgmap v44: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-06T03:24:31.403 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:31.403 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:31.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 188687 11.37 MB/sec execute 275 sec latency 33.985 ms 2024-04-06T03:24:31.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:31 smithi195 ceph-mon[53157]: pgmap v44: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-06T03:24:31.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:31.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:32.137 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:24:32.137 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (3m) 69s ago 13m 19.9M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:24:32.137 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 starting - - - - 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (12m) 53s ago 12m 6405k - 17.2.0 e1d6a67b021e cf0eca31da0e 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 unknown 69s ago 12m - - 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (7m) 69s ago 7m 40.4M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (7m) 52s ago 7m 15.4M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (2m) 69s ago 14m 478M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (111s) 53s ago 12m 448M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (92s) 69s ago 14m 29.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (59s) 53s ago 12m 27.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (6m) 69s ago 6m 271M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (6m) 52s ago 6m 81.1M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (3m) 69s ago 12m 9.98M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (2m) 53s ago 12m 19.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:24:32.138 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (11m) 69s ago 11m 570M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (11m) 69s ago 11m 488M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (10m) 69s ago 10m 357M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (10m) 69s ago 10m 492M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (9m) 53s ago 9m 532M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (9m) 53s ago 9m 363M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (9m) 53s ago 9m 547M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (8m) 53s ago 8m 481M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:24:32.139 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (119s) 69s ago 11m 53.6M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:24:32.416 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:32 smithi060 ceph-mon[67116]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:32.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:32 smithi195 ceph-mon[53157]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:32.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 189220 11.39 MB/sec execute 276 sec latency 33.952 ms 2024-04-06T03:24:32.832 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:24:32.832 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-06T03:24:32.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T03:24:32.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:24:32.834 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "mon" 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "4/23 daemons upgraded", 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading crash daemons", 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:24:33.420 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: pgmap v45: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.0 MiB/s wr, 492 op/s 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/400341368' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:33.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: pgmap v45: 97 pgs: 97 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.0 MiB/s wr, 492 op/s 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/400341368' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi195", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T03:24:33.609 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:33.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 189751 11.39 MB/sec execute 277 sec latency 52.911 ms 2024-04-06T03:24:34.112 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:24:34.112 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:24:34.112 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in unknown state 2024-04-06T03:24:34.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:34 smithi060 ceph-mon[67116]: Upgrade: Updating crash.smithi195 (2/2) 2024-04-06T03:24:34.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:34 smithi060 ceph-mon[67116]: Deploying daemon crash.smithi195 on smithi195 2024-04-06T03:24:34.442 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:34 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2744860777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:24:34.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 190134 11.40 MB/sec execute 278 sec latency 30.286 ms 2024-04-06T03:24:34.646 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:34 smithi195 ceph-mon[53157]: Upgrade: Updating crash.smithi195 (2/2) 2024-04-06T03:24:34.646 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:34 smithi195 ceph-mon[53157]: Deploying daemon crash.smithi195 on smithi195 2024-04-06T03:24:34.646 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:34 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/2744860777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:24:35.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:35 smithi060 ceph-mon[67116]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:35.337 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:35 smithi060 ceph-mon[67116]: pgmap v46: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-04-06T03:24:35.577 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:35 smithi195 ceph-mon[53157]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:24:35.577 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:35 smithi195 ceph-mon[53157]: pgmap v46: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-04-06T03:24:35.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 190629 11.40 MB/sec execute 279 sec latency 34.754 ms 2024-04-06T03:24:36.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 190892 11.39 MB/sec execute 280 sec latency 70.920 ms 2024-04-06T03:24:37.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:37 smithi195 ceph-mon[53157]: pgmap v47: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-06T03:24:37.486 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:37 smithi060 ceph-mon[67116]: pgmap v47: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-06T03:24:37.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 191081 11.37 MB/sec execute 281 sec latency 50.272 ms 2024-04-06T03:24:38.494 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:38.495 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:38.495 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:38.495 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:38 smithi060 ceph-mon[67116]: pgmap v48: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-06T03:24:38.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 191303 11.36 MB/sec execute 282 sec latency 51.970 ms 2024-04-06T03:24:38.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:38.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:38.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:38.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:38 smithi195 ceph-mon[53157]: pgmap v48: 97 pgs: 97 active+clean; 354 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-06T03:24:39.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 191561 11.34 MB/sec execute 283 sec latency 56.705 ms 2024-04-06T03:24:40.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 191898 11.32 MB/sec execute 284 sec latency 48.945 ms 2024-04-06T03:24:41.293 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:41 smithi060 ceph-mon[67116]: pgmap v49: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 806 op/s 2024-04-06T03:24:41.445 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:41 smithi195 ceph-mon[53157]: pgmap v49: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 806 op/s 2024-04-06T03:24:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 192184 11.30 MB/sec execute 285 sec latency 45.855 ms 2024-04-06T03:24:42.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 192361 11.28 MB/sec execute 286 sec latency 47.290 ms 2024-04-06T03:24:42.812 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:42.812 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:42.812 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:42 smithi060 ceph-mon[67116]: pgmap v50: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.5 MiB/s wr, 529 op/s 2024-04-06T03:24:42.812 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:42.970 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:42.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:42.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:42 smithi195 ceph-mon[53157]: pgmap v50: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.5 MiB/s wr, 529 op/s 2024-04-06T03:24:42.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 5 192745 11.29 MB/sec execute 287 sec latency 34.366 ms 2024-04-06T03:24:44.078 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:44.079 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:44.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:44.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 193222 11.27 MB/sec execute 288 sec latency 37.962 ms 2024-04-06T03:24:45.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:44 smithi060 ceph-mon[67116]: mgrmap e41: smithi060.jttwce(active, since 91s), standbys: smithi195.ultkot 2024-04-06T03:24:45.100 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:44 smithi060 ceph-mon[67116]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 700 op/s 2024-04-06T03:24:45.198 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:44 smithi195 ceph-mon[53157]: mgrmap e41: smithi060.jttwce(active, since 91s), standbys: smithi195.ultkot 2024-04-06T03:24:45.198 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:44 smithi195 ceph-mon[53157]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 700 op/s 2024-04-06T03:24:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 193567 11.28 MB/sec execute 289 sec latency 44.346 ms 2024-04-06T03:24:46.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 193903 11.28 MB/sec execute 290 sec latency 68.155 ms 2024-04-06T03:24:46.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:46 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:46.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:46 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:46.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:46 smithi060 ceph-mon[67116]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 384 op/s 2024-04-06T03:24:46.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:46 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:46.891 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:46 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:47.220 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:46 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:46 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:46 smithi195 ceph-mon[53157]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 384 op/s 2024-04-06T03:24:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:46 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:46 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:47.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 194304 11.27 MB/sec execute 291 sec latency 41.599 ms 2024-04-06T03:24:48.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:48 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:48.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:48 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:48.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 194590 11.26 MB/sec execute 292 sec latency 38.022 ms 2024-04-06T03:24:48.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:48 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:48.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:48 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:49.433 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:49 smithi060 ceph-mon[67116]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 384 op/s 2024-04-06T03:24:49.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 194851 11.25 MB/sec execute 293 sec latency 46.852 ms 2024-04-06T03:24:49.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:49 smithi195 ceph-mon[53157]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 384 op/s 2024-04-06T03:24:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 195416 11.26 MB/sec execute 294 sec latency 26.362 ms 2024-04-06T03:24:51.452 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:51 smithi060 ceph-mon[67116]: pgmap v54: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.2 MiB/s wr, 608 op/s 2024-04-06T03:24:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 195860 11.25 MB/sec execute 295 sec latency 29.592 ms 2024-04-06T03:24:51.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:51 smithi195 ceph-mon[53157]: pgmap v54: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.2 MiB/s wr, 608 op/s 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 196407 11.27 MB/sec execute 296 sec latency 47.162 ms 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: pgmap v55: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: pgmap v56: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 467 op/s 2024-04-06T03:24:53.463 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi060"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi060"}]': finished 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi195"}]: dispatch 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi195"}]': finished 2024-04-06T03:24:53.464 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: pgmap v55: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 395 op/s 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: pgmap v56: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 467 op/s 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi060"}]: dispatch 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi060"}]': finished 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi195"}]: dispatch 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi195"}]': finished 2024-04-06T03:24:53.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T03:24:53.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 197004 11.29 MB/sec execute 297 sec latency 32.751 ms 2024-04-06T03:24:54.219 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:54 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all crash 2024-04-06T03:24:54.219 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:54 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T03:24:54.219 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:54 smithi060 ceph-mon[67116]: Upgrade: osd.0 is safe to restart 2024-04-06T03:24:54.470 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:54 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all crash 2024-04-06T03:24:54.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:54 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T03:24:54.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:54 smithi195 ceph-mon[53157]: Upgrade: osd.0 is safe to restart 2024-04-06T03:24:54.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 197375 11.29 MB/sec execute 298 sec latency 45.415 ms 2024-04-06T03:24:55.225 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:55 smithi060 ceph-mon[67116]: pgmap v57: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:24:55.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 197822 11.28 MB/sec execute 299 sec latency 48.466 ms 2024-04-06T03:24:55.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:55 smithi195 ceph-mon[53157]: pgmap v57: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:24:56.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:56 smithi060 ceph-mon[67116]: Upgrade: Updating osd.0 2024-04-06T03:24:56.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:56.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:24:56.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:56.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:56 smithi060 ceph-mon[67116]: Deploying daemon osd.0 on smithi060 2024-04-06T03:24:56.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 198136 11.28 MB/sec execute 300 sec latency 64.737 ms 2024-04-06T03:24:56.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:56 smithi195 ceph-mon[53157]: Upgrade: Updating osd.0 2024-04-06T03:24:56.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:24:56.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:24:56.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:56.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:56 smithi195 ceph-mon[53157]: Deploying daemon osd.0 on smithi060 2024-04-06T03:24:57.262 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:57 smithi060 ceph-mon[67116]: pgmap v58: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:24:57.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 198460 11.29 MB/sec execute 301 sec latency 37.376 ms 2024-04-06T03:24:57.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:57 smithi195 ceph-mon[53157]: pgmap v58: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:24:58.267 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 198868 11.27 MB/sec execute 302 sec latency 62.303 ms 2024-04-06T03:24:58.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:59.277 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:24:59 smithi060 ceph-mon[67116]: pgmap v59: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:24:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 199254 11.26 MB/sec execute 303 sec latency 56.650 ms 2024-04-06T03:24:59.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:24:59 smithi195 ceph-mon[53157]: pgmap v59: 97 pgs: 97 active+clean; 356 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-06T03:25:00.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 199496 11.24 MB/sec execute 304 sec latency 43.245 ms 2024-04-06T03:25:01.087 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:00 smithi060 systemd[1]: Stopping Ceph osd.0 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:25:01.338 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:01 smithi060 ceph-mon[67116]: pgmap v60: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-04-06T03:25:01.592 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:01 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[35063]: 2024-04-06T03:25:01.465+0000 7f1124fe0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:25:01.592 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:01 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[35063]: 2024-04-06T03:25:01.465+0000 7f1124fe0700 -1 osd.0 73 *** Got signal Terminated *** 2024-04-06T03:25:01.592 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:01 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[35063]: 2024-04-06T03:25:01.465+0000 7f1124fe0700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:25:01.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 199870 11.25 MB/sec execute 305 sec latency 59.175 ms 2024-04-06T03:25:01.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:01 smithi195 ceph-mon[53157]: pgmap v60: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-04-06T03:25:02.497 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:02 smithi060 ceph-mon[67116]: osd.0 marked itself down and dead 2024-04-06T03:25:02.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 200149 11.23 MB/sec execute 306 sec latency 396.622 ms 2024-04-06T03:25:02.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:02 smithi195 ceph-mon[53157]: osd.0 marked itself down and dead 2024-04-06T03:25:03.247 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:02 smithi060 podman[73007]: 2024-04-06 03:25:02.941730542 +0000 UTC m=+1.760407350 container died 4a0e0dac96fec16c99caf7cb15f1c5e7e771e6e4d03fa972973e209a562fcd80 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T03:25:03.504 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:03 smithi060 ceph-mon[67116]: pgmap v61: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-04-06T03:25:03.504 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:03 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:25:03.504 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:03 smithi060 ceph-mon[67116]: osdmap e74: 8 total, 7 up, 8 in 2024-04-06T03:25:03.638 INFO:teuthology.orchestra.run.smithi060.stdout: 5 200273 11.20 MB/sec execute 307 sec latency 1058.760 ms 2024-04-06T03:25:03.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:03 smithi195 ceph-mon[53157]: pgmap v61: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-04-06T03:25:03.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:03 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:25:03.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:03 smithi195 ceph-mon[53157]: osdmap e74: 8 total, 7 up, 8 in 2024-04-06T03:25:04.007 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:03 smithi060 podman[73007]: 2024-04-06 03:25:03.84545636 +0000 UTC m=+2.664133180 container cleanup 4a0e0dac96fec16c99caf7cb15f1c5e7e771e6e4d03fa972973e209a562fcd80 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD) 2024-04-06T03:25:04.007 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:03 smithi060 bash[73007]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T03:25:04.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:04 smithi060 ceph-mon[67116]: osdmap e75: 8 total, 7 up, 8 in 2024-04-06T03:25:04.499 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:04 smithi060 podman[73023]: 2024-04-06 03:25:04.314268594 +0000 UTC m=+1.367430918 container remove 4a0e0dac96fec16c99caf7cb15f1c5e7e771e6e4d03fa972973e209a562fcd80 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, release=754, vendor=Red Hat, Inc., vcs-type=git, GIT_CLEAN=True, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-06T03:25:04.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 200612 11.20 MB/sec execute 308 sec latency 127.282 ms 2024-04-06T03:25:04.639 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:25:04.720 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:04 smithi195 ceph-mon[53157]: osdmap e75: 8 total, 7 up, 8 in 2024-04-06T03:25:05.036 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:04 smithi060 podman[73136]: 2024-04-06 03:25:04.763791748 +0000 UTC m=+0.022741863 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:05.036 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:04 smithi060 podman[73136]: 2024-04-06 03:25:04.969314207 +0000 UTC m=+0.228264295 container create 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T03:25:05.297 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:05 smithi060 ceph-mon[67116]: pgmap v64: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 628 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:05.297 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:05 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 220/1647 objects degraded (13.358%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:05.297 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:05 smithi060 podman[73136]: 2024-04-06 03:25:05.294823923 +0000 UTC m=+0.553774022 container init 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-06T03:25:05.554 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:05 smithi060 podman[73136]: 2024-04-06 03:25:05.299769853 +0000 UTC m=+0.558719944 container start 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:25:05.554 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:05 smithi060 podman[73136]: 2024-04-06 03:25:05.456348196 +0000 UTC m=+0.715298309 container attach 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (4m) 19s ago 13m 17.9M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (35s) 19s ago 13m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (28s) 23s ago 12m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 19s ago 13m - - 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (7m) 19s ago 7m 40.2M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (7m) 23s ago 7m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (3m) 19s ago 14m 501M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:25:05.631 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (2m) 23s ago 12m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (2m) 19s ago 15m 43.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (93s) 23s ago 12m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (7m) 19s ago 7m 274M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (7m) 23s ago 7m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (4m) 19s ago 13m 9717k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (3m) 23s ago 12m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (12m) 19s ago 12m 401M 1937M 17.2.0 e1d6a67b021e 4a0e0dac96fe 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (11m) 19s ago 11m 583M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (11m) 19s ago 11m 464M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (10m) 19s ago 10m 592M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (10m) 23s ago 10m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (9m) 23s ago 9m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (9m) 23s ago 9m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:25:05.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (9m) 23s ago 9m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:25:05.633 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (2m) 19s ago 12m 60.6M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:25:05.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 200821 11.18 MB/sec execute 309 sec latency 49.672 ms 2024-04-06T03:25:05.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:05 smithi195 ceph-mon[53157]: pgmap v64: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 628 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:05.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:05 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 220/1647 objects degraded (13.358%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:05.813 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:05 smithi060 podman[73136]: 2024-04-06 03:25:05.706224937 +0000 UTC m=+0.965175033 container died 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, ceph=True) 2024-04-06T03:25:06.108 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-06T03:25:06.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T03:25:06.110 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:25:06.110 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:25:06.322 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:06 smithi060 ceph-mon[67116]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:06.322 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:06 smithi060 ceph-mon[67116]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:06.322 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:06 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2893728470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:06.558 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "6/23 daemons upgraded", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:25:06.559 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:25:06.560 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:25:06.577 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:06 smithi060 podman[73207]: 2024-04-06 03:25:06.569372285 +0000 UTC m=+0.856537456 container remove 5ed0ce74e51dbef41785475c7c29ae1aaef43e084e9c92535c8659c2b2655f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-06T03:25:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 201085 11.18 MB/sec execute 310 sec latency 72.319 ms 2024-04-06T03:25:06.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:06 smithi195 ceph-mon[53157]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:06.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:06 smithi195 ceph-mon[53157]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:06.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:06 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/2893728470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:06.840 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:06 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-04-06T03:25:06.840 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:06 smithi060 systemd[1]: Stopped Ceph osd.0 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:25:06.840 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:06 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.0.service: Consumed 3min 46.917s CPU time. 2024-04-06T03:25:07.093 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:06 smithi060 systemd[1]: Starting Ceph osd.0 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 220/1647 objects degraded (13.358%), 28 pgs degraded 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout: osd.0 (root=default,host=smithi060) is down 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 220/1647 objects degraded (13.358%), 28 pgs degraded 2024-04-06T03:25:07.105 INFO:teuthology.orchestra.run.smithi060.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.f is active+undersized+degraded, acting [4,2] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-04-06T03:25:07.106 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-04-06T03:25:07.107 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-04-06T03:25:07.350 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:07 smithi060 ceph-mon[67116]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:07.350 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:07 smithi060 ceph-mon[67116]: pgmap v65: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 628 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:07.350 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:07 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/559869301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:25:07.351 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:07 smithi060 podman[73405]: 2024-04-06 03:25:07.093501303 +0000 UTC m=+0.025387187 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:07.351 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:07 smithi060 podman[73405]: 2024-04-06 03:25:07.27186345 +0000 UTC m=+0.203749322 container create f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-06T03:25:07.608 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:07 smithi060 podman[73405]: 2024-04-06 03:25:07.57563445 +0000 UTC m=+0.507520320 container init f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:25:07.608 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:07 smithi060 podman[73405]: 2024-04-06 03:25:07.580280954 +0000 UTC m=+0.512166825 container start f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-06T03:25:07.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 201429 11.16 MB/sec execute 311 sec latency 54.322 ms 2024-04-06T03:25:07.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:07 smithi195 ceph-mon[53157]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:07.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:07 smithi195 ceph-mon[53157]: pgmap v65: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.2 MiB/s wr, 628 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:07.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:07 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/559869301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:25:07.859 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:07 smithi060 podman[73405]: 2024-04-06 03:25:07.692746681 +0000 UTC m=+0.624632575 container attach f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:25:08.115 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 bash[73405]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-06T03:25:08.116 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T03:25:08.117 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate[73423]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-06T03:25:08.366 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:08 smithi060 ceph-mon[67116]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:08.366 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:08 smithi060 podman[73405]: 2024-04-06 03:25:08.126779677 +0000 UTC m=+1.058665553 container died f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-06T03:25:08.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 201696 11.15 MB/sec execute 312 sec latency 75.496 ms 2024-04-06T03:25:08.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:08 smithi195 ceph-mon[53157]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:09.273 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:09 smithi060 podman[73559]: 2024-04-06 03:25:09.092933918 +0000 UTC m=+0.960729476 container remove f4d90b2cfa479c0bbb6d37b5e09e1d21da9d3c4fea3d284b759466ec5b89e73d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-06T03:25:09.524 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:09 smithi060 ceph-mon[67116]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.4 MiB/s wr, 260 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:09.524 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:09 smithi060 podman[73618]: 2024-04-06 03:25:09.274456315 +0000 UTC m=+0.035281248 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:09.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 202352 11.18 MB/sec execute 313 sec latency 33.169 ms 2024-04-06T03:25:09.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:09 smithi195 ceph-mon[53157]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.4 MiB/s wr, 260 op/s; 220/1647 objects degraded (13.358%) 2024-04-06T03:25:10.028 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:09 smithi060 podman[73618]: 2024-04-06 03:25:09.786589045 +0000 UTC m=+0.547413996 container create dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-06T03:25:10.279 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 podman[73618]: 2024-04-06 03:25:10.182741473 +0000 UTC m=+0.943566395 container init dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:25:10.279 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 podman[73618]: 2024-04-06 03:25:10.198567545 +0000 UTC m=+0.959392468 container start dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-06T03:25:10.279 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 ceph-osd[73636]: -- 172.21.15.60:0/1384167717 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e84c8d5680 con 0x55e84c89ec00 2024-04-06T03:25:10.530 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:10 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 236/1692 objects degraded (13.948%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:10.530 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 bash[73618]: dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 2024-04-06T03:25:10.530 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 systemd[1]: Started Ceph osd.0 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:25:10.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 203018 11.18 MB/sec execute 314 sec latency 19.726 ms 2024-04-06T03:25:10.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:10 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 236/1692 objects degraded (13.948%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:11.032 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:10 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:25:10.789+0000 7f107f6837c0 -1 Falling back to public interface 2024-04-06T03:25:11.539 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:11 smithi060 ceph-mon[67116]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 525 op/s; 236/1692 objects degraded (13.948%) 2024-04-06T03:25:11.539 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:11.539 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:11.539 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 203670 11.21 MB/sec execute 315 sec latency 34.197 ms 2024-04-06T03:25:11.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:11 smithi195 ceph-mon[53157]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 525 op/s; 236/1692 objects degraded (13.948%) 2024-04-06T03:25:11.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:11.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:11.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:12.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 204147 11.22 MB/sec execute 316 sec latency 48.861 ms 2024-04-06T03:25:13.576 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:13 smithi060 ceph-mon[67116]: pgmap v68: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 421 op/s; 236/1692 objects degraded (13.948%) 2024-04-06T03:25:13.576 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:13.576 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:13.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 204556 11.23 MB/sec execute 317 sec latency 32.470 ms 2024-04-06T03:25:13.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:13 smithi195 ceph-mon[53157]: pgmap v68: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 351 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 421 op/s; 236/1692 objects degraded (13.948%) 2024-04-06T03:25:13.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:13.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:14.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 204939 11.21 MB/sec execute 318 sec latency 47.315 ms 2024-04-06T03:25:15.610 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:15 smithi060 ceph-mon[67116]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.5 MiB/s wr, 757 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: 5 205294 11.21 MB/sec execute 319 sec latency 33.035 ms 2024-04-06T03:25:15.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:15 smithi195 ceph-mon[53157]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 7.5 MiB/s wr, 757 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:16.619 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:16 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 238/1794 objects degraded (13.266%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:16.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 205674 11.23 MB/sec execute 320 sec latency 34.964 ms 2024-04-06T03:25:16.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:16 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 238/1794 objects degraded (13.266%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:17.121 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:16 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:25:16.902+0000 7f107f6837c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-06T03:25:17.372 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:17 smithi060 ceph-mon[67116]: pgmap v70: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.3 MiB/s wr, 519 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 206081 11.21 MB/sec execute 321 sec latency 35.425 ms 2024-04-06T03:25:17.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:17 smithi195 ceph-mon[53157]: pgmap v70: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.3 MiB/s wr, 519 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:18.386 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:18 smithi060 ceph-mon[67116]: pgmap v71: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.3 MiB/s wr, 519 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:18.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 206479 11.20 MB/sec execute 322 sec latency 40.413 ms 2024-04-06T03:25:18.647 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:18 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:25:18.488+0000 7f107f6837c0 -1 osd.0 73 log_to_monitors true 2024-04-06T03:25:18.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:18 smithi195 ceph-mon[53157]: pgmap v71: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.3 MiB/s wr, 519 op/s; 238/1794 objects degraded (13.266%) 2024-04-06T03:25:19.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:19 smithi060 ceph-mon[67116]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:25:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 206956 11.22 MB/sec execute 323 sec latency 31.205 ms 2024-04-06T03:25:19.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:19 smithi195 ceph-mon[53157]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:25:19.839 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:25:19 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:25:19.719+0000 7f106cad3700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:25:20.424 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: osdmap e76: 8 total, 7 up, 8 in 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' 2024-04-06T03:25:20.425 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:20 smithi060 ceph-mon[67116]: pgmap v73: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.1 MiB/s wr, 702 op/s; 225/1722 objects degraded (13.066%) 2024-04-06T03:25:20.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 207463 11.21 MB/sec execute 324 sec latency 40.270 ms 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: osdmap e76: 8 total, 7 up, 8 in 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: from='osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873]' entity='osd.0' 2024-04-06T03:25:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:20 smithi195 ceph-mon[53157]: pgmap v73: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.1 MiB/s wr, 702 op/s; 225/1722 objects degraded (13.066%) 2024-04-06T03:25:21.642 INFO:teuthology.orchestra.run.smithi060.stdout: 5 207616 11.19 MB/sec execute 325 sec latency 873.362 ms 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 225/1722 objects degraded (13.066%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873] boot 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: osdmap e77: 8 total, 8 up, 8 in 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 225/1722 objects degraded (13.066%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: osd.0 [v2:172.21.15.60:6802/568090873,v1:172.21.15.60:6803/568090873] boot 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: osdmap e77: 8 total, 8 up, 8 in 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:21.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 207629 11.16 MB/sec execute 326 sec latency 1873.390 ms 2024-04-06T03:25:22.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:22 smithi195 ceph-mon[53157]: osdmap e78: 8 total, 8 up, 8 in 2024-04-06T03:25:22.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:22 smithi195 ceph-mon[53157]: pgmap v76: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 225/1722 objects degraded (13.066%) 2024-04-06T03:25:22.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:22 smithi060 ceph-mon[67116]: osdmap e78: 8 total, 8 up, 8 in 2024-04-06T03:25:22.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:22 smithi060 ceph-mon[67116]: pgmap v76: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 225/1722 objects degraded (13.066%) 2024-04-06T03:25:23.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 207786 11.15 MB/sec execute 327 sec latency 2319.441 ms 2024-04-06T03:25:23.796 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:23 smithi060 ceph-mon[67116]: osdmap e79: 8 total, 8 up, 8 in 2024-04-06T03:25:23.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:23 smithi195 ceph-mon[53157]: osdmap e79: 8 total, 8 up, 8 in 2024-04-06T03:25:24.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 208218 11.15 MB/sec execute 328 sec latency 232.177 ms 2024-04-06T03:25:24.801 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:24 smithi060 ceph-mon[67116]: osdmap e80: 8 total, 8 up, 8 in 2024-04-06T03:25:24.801 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:24 smithi060 ceph-mon[67116]: pgmap v79: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized, 77 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 481 op/s; 238/1533 objects degraded (15.525%); 1.4 MiB/s, 12 objects/s recovering 2024-04-06T03:25:25.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:24 smithi195 ceph-mon[53157]: osdmap e80: 8 total, 8 up, 8 in 2024-04-06T03:25:25.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:24 smithi195 ceph-mon[53157]: pgmap v79: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized, 77 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 481 op/s; 238/1533 objects degraded (15.525%); 1.4 MiB/s, 12 objects/s recovering 2024-04-06T03:25:25.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 208479 11.13 MB/sec execute 329 sec latency 844.770 ms 2024-04-06T03:25:25.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: osdmap e81: 8 total, 8 up, 8 in 2024-04-06T03:25:25.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:25.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:25.817 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: osdmap e82: 8 total, 8 up, 8 in 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:25.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:25 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: osdmap e81: 8 total, 8 up, 8 in 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:25:26.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: osdmap e82: 8 total, 8 up, 8 in 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:25 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:26.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 208681 11.11 MB/sec execute 330 sec latency 773.872 ms 2024-04-06T03:25:26.914 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: pgmap v81: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 510 op/s; 214/1527 objects degraded (14.014%); 1.5 MiB/s, 12 objects/s recovering 2024-04-06T03:25:26.914 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: pgmap v82: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 593 op/s; 214/1527 objects degraded (14.014%); 1.7 MiB/s, 14 objects/s recovering 2024-04-06T03:25:26.914 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:26.915 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T03:25:26.915 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 214/1527 objects degraded (14.014%), 9 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:26.915 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:26 smithi060 ceph-mon[67116]: osdmap e83: 8 total, 8 up, 8 in 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: pgmap v81: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.0 MiB/s wr, 510 op/s; 214/1527 objects degraded (14.014%); 1.5 MiB/s, 12 objects/s recovering 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: pgmap v82: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 593 op/s; 214/1527 objects degraded (14.014%); 1.7 MiB/s, 14 objects/s recovering 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 214/1527 objects degraded (14.014%), 9 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:27.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:26 smithi195 ceph-mon[53157]: osdmap e83: 8 total, 8 up, 8 in 2024-04-06T03:25:27.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 208926 11.11 MB/sec execute 331 sec latency 59.473 ms 2024-04-06T03:25:28.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:28.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:28.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:28 smithi060 ceph-mon[67116]: osdmap e84: 8 total, 8 up, 8 in 2024-04-06T03:25:28.640 INFO:teuthology.orchestra.run.smithi060.stdout: 5 209127 11.09 MB/sec execute 332 sec latency 493.597 ms 2024-04-06T03:25:28.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:28.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:28.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:28 smithi195 ceph-mon[53157]: osdmap e84: 8 total, 8 up, 8 in 2024-04-06T03:25:29.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:29 smithi060 ceph-mon[67116]: pgmap v85: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 35 KiB/s wr, 6 op/s; 214/1527 objects degraded (14.014%) 2024-04-06T03:25:29.496 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:29 smithi060 ceph-mon[67116]: osdmap e85: 8 total, 8 up, 8 in 2024-04-06T03:25:29.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 209431 11.07 MB/sec execute 333 sec latency 96.562 ms 2024-04-06T03:25:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:29 smithi195 ceph-mon[53157]: pgmap v85: 97 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 85 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 35 KiB/s wr, 6 op/s; 214/1527 objects degraded (14.014%) 2024-04-06T03:25:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:29 smithi195 ceph-mon[53157]: osdmap e85: 8 total, 8 up, 8 in 2024-04-06T03:25:30.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 209736 11.05 MB/sec execute 334 sec latency 195.226 ms 2024-04-06T03:25:30.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:30 smithi060 ceph-mon[67116]: pgmap v88: 97 pgs: 1 active+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 530 op/s; 92/1581 objects degraded (5.819%); 0 B/s, 38 objects/s recovering 2024-04-06T03:25:30.763 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:30 smithi060 ceph-mon[67116]: osdmap e86: 8 total, 8 up, 8 in 2024-04-06T03:25:30.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:30 smithi195 ceph-mon[53157]: pgmap v88: 97 pgs: 1 active+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 530 op/s; 92/1581 objects degraded (5.819%); 0 B/s, 38 objects/s recovering 2024-04-06T03:25:30.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:30 smithi195 ceph-mon[53157]: osdmap e86: 8 total, 8 up, 8 in 2024-04-06T03:25:31.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 209865 11.03 MB/sec execute 335 sec latency 943.626 ms 2024-04-06T03:25:31.894 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:31 smithi060 ceph-mon[67116]: osdmap e87: 8 total, 8 up, 8 in 2024-04-06T03:25:31.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:31 smithi195 ceph-mon[53157]: osdmap e87: 8 total, 8 up, 8 in 2024-04-06T03:25:32.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 210152 11.04 MB/sec execute 336 sec latency 971.104 ms 2024-04-06T03:25:32.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:32 smithi060 ceph-mon[67116]: pgmap v91: 97 pgs: 1 active+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 530 op/s; 92/1581 objects degraded (5.819%); 0 B/s, 38 objects/s recovering 2024-04-06T03:25:32.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:32 smithi060 ceph-mon[67116]: osdmap e88: 8 total, 8 up, 8 in 2024-04-06T03:25:32.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:32 smithi195 ceph-mon[53157]: pgmap v91: 97 pgs: 1 active+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 530 op/s; 92/1581 objects degraded (5.819%); 0 B/s, 38 objects/s recovering 2024-04-06T03:25:32.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:32 smithi195 ceph-mon[53157]: osdmap e88: 8 total, 8 up, 8 in 2024-04-06T03:25:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 210421 11.03 MB/sec execute 337 sec latency 57.074 ms 2024-04-06T03:25:33.675 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:33 smithi060 ceph-mon[67116]: osdmap e89: 8 total, 8 up, 8 in 2024-04-06T03:25:33.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:33 smithi195 ceph-mon[53157]: osdmap e89: 8 total, 8 up, 8 in 2024-04-06T03:25:34.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 210803 11.03 MB/sec execute 338 sec latency 43.212 ms 2024-04-06T03:25:34.688 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:34 smithi060 ceph-mon[67116]: pgmap v94: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 36/1614 objects degraded (2.230%); 0 B/s, 13 objects/s recovering 2024-04-06T03:25:34.688 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:34 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 36/1614 objects degraded (2.230%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:34.688 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:34 smithi060 ceph-mon[67116]: osdmap e90: 8 total, 8 up, 8 in 2024-04-06T03:25:34.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:34 smithi195 ceph-mon[53157]: pgmap v94: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 36/1614 objects degraded (2.230%); 0 B/s, 13 objects/s recovering 2024-04-06T03:25:34.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:34 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 36/1614 objects degraded (2.230%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:34.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:34 smithi195 ceph-mon[53157]: osdmap e90: 8 total, 8 up, 8 in 2024-04-06T03:25:35.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 211119 11.02 MB/sec execute 339 sec latency 59.505 ms 2024-04-06T03:25:35.960 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:35 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/1614 objects degraded (2.230%), 2 pgs degraded) 2024-04-06T03:25:36.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:35 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/1614 objects degraded (2.230%), 2 pgs degraded) 2024-04-06T03:25:36.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 211457 11.01 MB/sec execute 340 sec latency 40.181 ms 2024-04-06T03:25:36.730 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:36 smithi060 ceph-mon[67116]: pgmap v96: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 388 op/s; 0 B/s, 21 objects/s recovering 2024-04-06T03:25:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:36 smithi195 ceph-mon[53157]: pgmap v96: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 388 op/s; 0 B/s, 21 objects/s recovering 2024-04-06T03:25:37.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 211774 11.00 MB/sec execute 341 sec latency 56.759 ms 2024-04-06T03:25:37.719 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:25:38.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 212064 10.99 MB/sec execute 342 sec latency 52.473 ms 2024-04-06T03:25:38.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:38 smithi060 ceph-mon[67116]: pgmap v97: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.3 MiB/s wr, 317 op/s; 0 B/s, 17 objects/s recovering 2024-04-06T03:25:38.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:38 smithi060 ceph-mon[67116]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:38.767 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:38 smithi060 ceph-mon[67116]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:38.827 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:25:38.827 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (4m) 19s ago 14m 17.9M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:25:38.827 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (68s) 19s ago 14m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:25:38.827 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (62s) 57s ago 13m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:25:38.827 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 19s ago 13m - - 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (8m) 19s ago 8m 40.2M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (8m) 57s ago 8m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (3m) 19s ago 15m 503M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (2m) 57s ago 13m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (2m) 19s ago 15m 47.0M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (2m) 57s ago 13m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (7m) 19s ago 7m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (7m) 57s ago 7m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (4m) 19s ago 13m 9139k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (4m) 57s ago 13m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (29s) 19s ago 12m 12.1M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (12m) 19s ago 12m 358M 1937M 17.2.0 e1d6a67b021e 6aabf37aefc1 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (11m) 19s ago 11m 492M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:25:38.828 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (11m) 19s ago 11m 354M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:25:38.829 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (10m) 57s ago 10m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:25:38.829 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (10m) 57s ago 10m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:25:38.829 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (10m) 57s ago 10m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:25:38.829 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (9m) 57s ago 9m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:25:38.829 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (3m) 19s ago 12m 60.7M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:25:38.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:38 smithi195 ceph-mon[53157]: pgmap v97: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.3 MiB/s wr, 317 op/s; 0 B/s, 17 objects/s recovering 2024-04-06T03:25:38.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:38 smithi195 ceph-mon[53157]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:38.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:38 smithi195 ceph-mon[53157]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:39.465 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:25:39.466 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:25:39.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 212353 10.98 MB/sec execute 343 sec latency 64.162 ms 2024-04-06T03:25:39.789 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:39 smithi060 ceph-mon[67116]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:39.789 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:39 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2795021817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "7/23 daemons upgraded", 2024-04-06T03:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:25:40.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:39 smithi195 ceph-mon[53157]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:40.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:39 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/2795021817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:25:40.542 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:25:40.542 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:25:40.542 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:25:40.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 212624 10.98 MB/sec execute 344 sec latency 65.269 ms 2024-04-06T03:25:40.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:40 smithi060 ceph-mon[67116]: pgmap v98: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 0 B/s, 13 objects/s recovering 2024-04-06T03:25:40.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:40 smithi060 ceph-mon[67116]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:40.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:40 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/74767437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:25:40.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:41.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:40 smithi195 ceph-mon[53157]: pgmap v98: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 0 B/s, 13 objects/s recovering 2024-04-06T03:25:41.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:40 smithi195 ceph-mon[53157]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:25:41.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:40 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/74767437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:25:41.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: 5 212990 10.96 MB/sec execute 345 sec latency 54.088 ms 2024-04-06T03:25:41.773 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:41 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:41.774 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:41 smithi060 ceph-mon[67116]: Upgrade: osd.1 is safe to restart 2024-04-06T03:25:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:41 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T03:25:42.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:41 smithi195 ceph-mon[53157]: Upgrade: osd.1 is safe to restart 2024-04-06T03:25:42.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 213306 10.95 MB/sec execute 346 sec latency 47.036 ms 2024-04-06T03:25:43.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:43 smithi060 ceph-mon[67116]: pgmap v99: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 487 op/s; 0 B/s, 11 objects/s recovering 2024-04-06T03:25:43.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:43.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 213667 10.96 MB/sec execute 347 sec latency 42.643 ms 2024-04-06T03:25:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:43 smithi195 ceph-mon[53157]: pgmap v99: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 487 op/s; 0 B/s, 11 objects/s recovering 2024-04-06T03:25:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: Upgrade: Updating osd.1 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: Deploying daemon osd.1 on smithi060 2024-04-06T03:25:44.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:44 smithi060 ceph-mon[67116]: pgmap v100: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 466 op/s; 0 B/s, 4 objects/s recovering 2024-04-06T03:25:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 213994 10.96 MB/sec execute 348 sec latency 60.946 ms 2024-04-06T03:25:44.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: Upgrade: Updating osd.1 2024-04-06T03:25:44.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:44.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:25:44.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:44.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: Deploying daemon osd.1 on smithi060 2024-04-06T03:25:44.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:44 smithi195 ceph-mon[53157]: pgmap v100: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 466 op/s; 0 B/s, 4 objects/s recovering 2024-04-06T03:25:45.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 214462 10.96 MB/sec execute 349 sec latency 47.465 ms 2024-04-06T03:25:46.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 214829 10.95 MB/sec execute 350 sec latency 34.752 ms 2024-04-06T03:25:46.778 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:46 smithi060 ceph-mon[67116]: pgmap v101: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 407 op/s; 0 B/s, 4 objects/s recovering 2024-04-06T03:25:46.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:46 smithi195 ceph-mon[53157]: pgmap v101: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 407 op/s; 0 B/s, 4 objects/s recovering 2024-04-06T03:25:47.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 215218 10.94 MB/sec execute 351 sec latency 42.738 ms 2024-04-06T03:25:48.538 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:48 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:48.538 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:48 smithi060 ceph-mon[67116]: pgmap v102: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 390 op/s 2024-04-06T03:25:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 215653 10.95 MB/sec execute 352 sec latency 57.150 ms 2024-04-06T03:25:48.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:48 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:48.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:48 smithi195 ceph-mon[53157]: pgmap v102: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 390 op/s 2024-04-06T03:25:49.041 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:48 smithi060 systemd[1]: Stopping Ceph osd.1 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:25:49.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 215965 10.95 MB/sec execute 353 sec latency 205.540 ms 2024-04-06T03:25:49.700 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:49 smithi060 ceph-mon[67116]: osd.1 marked itself down and dead 2024-04-06T03:25:49.700 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:49 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:25:49.701 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:49 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[38196]: 2024-04-06T03:25:49.433+0000 7f19e9116700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:25:49.701 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:49 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[38196]: 2024-04-06T03:25:49.433+0000 7f19e9116700 -1 osd.1 90 *** Got signal Terminated *** 2024-04-06T03:25:49.701 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:49 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[38196]: 2024-04-06T03:25:49.433+0000 7f19e9116700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:25:49.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:49 smithi195 ceph-mon[53157]: osd.1 marked itself down and dead 2024-04-06T03:25:49.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:49 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:25:50.497 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:50 smithi060 podman[75722]: 2024-04-06 03:25:50.184848026 +0000 UTC m=+0.980658175 container died 6aabf37aefc1459137f1a940be5b879f5bd728f2c31677e21f833bd86f35d853 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, com.redhat.component=centos-stream-container, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, name=centos-stream, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., release=754, version=8, GIT_BRANCH=HEAD, architecture=x86_64) 2024-04-06T03:25:50.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 216023 10.92 MB/sec execute 354 sec latency 1053.483 ms 2024-04-06T03:25:50.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:50 smithi060 ceph-mon[67116]: osdmap e91: 8 total, 7 up, 8 in 2024-04-06T03:25:50.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:50 smithi060 ceph-mon[67116]: pgmap v104: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.3 MiB/s wr, 523 op/s 2024-04-06T03:25:50.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:50 smithi060 ceph-mon[67116]: osdmap e92: 8 total, 7 up, 8 in 2024-04-06T03:25:50.751 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:50 smithi060 podman[75722]: 2024-04-06 03:25:50.628519101 +0000 UTC m=+1.424329244 container cleanup 6aabf37aefc1459137f1a940be5b879f5bd728f2c31677e21f833bd86f35d853 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , version=8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., GIT_CLEAN=True) 2024-04-06T03:25:50.751 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:50 smithi060 bash[75722]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T03:25:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:50 smithi195 ceph-mon[53157]: osdmap e91: 8 total, 7 up, 8 in 2024-04-06T03:25:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:50 smithi195 ceph-mon[53157]: pgmap v104: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.3 MiB/s wr, 523 op/s 2024-04-06T03:25:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:50 smithi195 ceph-mon[53157]: osdmap e92: 8 total, 7 up, 8 in 2024-04-06T03:25:51.265 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:51 smithi060 podman[75742]: 2024-04-06 03:25:51.064537409 +0000 UTC m=+0.882784434 container remove 6aabf37aefc1459137f1a940be5b879f5bd728f2c31677e21f833bd86f35d853 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T03:25:51.521 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:51 smithi060 podman[75823]: 2024-04-06 03:25:51.398808622 +0000 UTC m=+0.031877807 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 216449 10.91 MB/sec execute 355 sec latency 40.329 ms 2024-04-06T03:25:51.772 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:51 smithi060 podman[75823]: 2024-04-06 03:25:51.524264196 +0000 UTC m=+0.157333371 container create b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-06T03:25:52.023 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:51 smithi060 podman[75823]: 2024-04-06 03:25:51.900896924 +0000 UTC m=+0.533966112 container init b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:25:52.023 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:51 smithi060 podman[75823]: 2024-04-06 03:25:51.910984439 +0000 UTC m=+0.544053618 container start b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-06T03:25:52.275 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:52 smithi060 podman[75823]: 2024-04-06 03:25:52.079225152 +0000 UTC m=+0.712294334 container attach b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_BRANCH=HEAD) 2024-04-06T03:25:52.526 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:52 smithi060 podman[75823]: 2024-04-06 03:25:52.396374954 +0000 UTC m=+1.029444129 container died b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-06T03:25:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 216885 10.91 MB/sec execute 356 sec latency 31.052 ms 2024-04-06T03:25:52.777 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:52 smithi060 ceph-mon[67116]: pgmap v106: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 390 op/s 2024-04-06T03:25:52.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:52 smithi195 ceph-mon[53157]: pgmap v106: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 390 op/s 2024-04-06T03:25:53.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 217269 10.92 MB/sec execute 357 sec latency 43.840 ms 2024-04-06T03:25:53.797 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:53 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 172/1731 objects degraded (9.936%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:53.797 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:53 smithi060 podman[75848]: 2024-04-06 03:25:53.580053496 +0000 UTC m=+1.167860208 container remove b6dfd36628867be1014f31fd36e88ada50445f98d7d8696be8b498c80341296c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-06T03:25:53.797 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:53 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-04-06T03:25:53.797 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:53 smithi060 systemd[1]: Stopped Ceph osd.1 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:25:53.797 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:53 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.1.service: Consumed 4min 2.005s CPU time. 2024-04-06T03:25:53.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:53 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 172/1731 objects degraded (9.936%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:25:54.057 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:53 smithi060 systemd[1]: Starting Ceph osd.1 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:25:54.312 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:54 smithi060 podman[75940]: 2024-04-06 03:25:54.134086117 +0000 UTC m=+0.022171434 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:54.312 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:54 smithi060 podman[75940]: 2024-04-06 03:25:54.256948543 +0000 UTC m=+0.145033880 container create 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, org.label-schema.license=GPLv2) 2024-04-06T03:25:54.570 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:54 smithi060 podman[75940]: 2024-04-06 03:25:54.543540082 +0000 UTC m=+0.431625378 container init 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T03:25:54.571 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:54 smithi060 podman[75940]: 2024-04-06 03:25:54.550447678 +0000 UTC m=+0.438532968 container start 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:25:54.653 INFO:teuthology.orchestra.run.smithi060.stdout: 5 217610 10.92 MB/sec execute 358 sec latency 39.787 ms 2024-04-06T03:25:54.838 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:54 smithi060 ceph-mon[67116]: pgmap v107: 97 pgs: 22 active+undersized+degraded, 1 stale+active+clean, 13 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.7 MiB/s wr, 554 op/s; 172/1731 objects degraded (9.936%) 2024-04-06T03:25:54.839 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:54 smithi060 podman[75940]: 2024-04-06 03:25:54.6945866 +0000 UTC m=+0.582671900 container attach 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:25:54.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:54 smithi195 ceph-mon[53157]: pgmap v107: 97 pgs: 22 active+undersized+degraded, 1 stale+active+clean, 13 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.7 MiB/s wr, 554 op/s; 172/1731 objects degraded (9.936%) 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 bash[75940]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T03:25:55.348 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T03:25:55.349 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T03:25:55.349 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-06T03:25:55.349 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T03:25:55.349 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate[75954]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-06T03:25:55.349 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:55 smithi060 podman[76079]: 2024-04-06 03:25:55.223562226 +0000 UTC m=+0.021953696 container died 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-06T03:25:55.652 INFO:teuthology.orchestra.run.smithi060.stdout: 5 217991 10.92 MB/sec execute 359 sec latency 43.460 ms 2024-04-06T03:25:56.379 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 podman[76079]: 2024-04-06 03:25:56.234483916 +0000 UTC m=+1.032875368 container remove 3c2b99a805cef1979f098e606d14f980bfe500a555918020dc45b7da330fa1c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-06T03:25:56.617 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 podman[76136]: 2024-04-06 03:25:56.4091797 +0000 UTC m=+0.034837416 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:25:56.617 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 podman[76136]: 2024-04-06 03:25:56.544829769 +0000 UTC m=+0.170487475 container create 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-06T03:25:56.635 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:56 smithi060 ceph-mon[67116]: pgmap v108: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.6 MiB/s wr, 655 op/s; 191/1731 objects degraded (11.034%) 2024-04-06T03:25:56.656 INFO:teuthology.orchestra.run.smithi060.stdout: 5 218327 10.91 MB/sec execute 360 sec latency 74.169 ms 2024-04-06T03:25:56.886 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 podman[76136]: 2024-04-06 03:25:56.855937033 +0000 UTC m=+0.481594742 container init 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-04-06T03:25:56.886 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 podman[76136]: 2024-04-06 03:25:56.865375553 +0000 UTC m=+0.491033265 container start 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-06T03:25:56.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:56 smithi195 ceph-mon[53157]: pgmap v108: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.6 MiB/s wr, 655 op/s; 191/1731 objects degraded (11.034%) 2024-04-06T03:25:57.147 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:56 smithi060 ceph-osd[76154]: -- 172.21.15.60:0/815403771 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559cd4c37680 con 0x559cd4c00c00 2024-04-06T03:25:57.147 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:57 smithi060 bash[76136]: 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 2024-04-06T03:25:57.147 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:57 smithi060 systemd[1]: Started Ceph osd.1 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:25:57.648 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:25:57 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:25:57.443+0000 7fbcd10dd7c0 -1 Falling back to public interface 2024-04-06T03:25:57.656 INFO:teuthology.orchestra.run.smithi060.stdout: 5 218656 10.90 MB/sec execute 361 sec latency 34.829 ms 2024-04-06T03:25:58.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:58.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:58.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:58.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:25:58.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 219140 10.91 MB/sec execute 362 sec latency 31.443 ms 2024-04-06T03:25:59.423 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:25:59 smithi060 ceph-mon[67116]: pgmap v109: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 284 op/s; 191/1731 objects degraded (11.034%) 2024-04-06T03:25:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 219655 10.93 MB/sec execute 363 sec latency 33.568 ms 2024-04-06T03:25:59.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:25:59 smithi195 ceph-mon[53157]: pgmap v109: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 284 op/s; 191/1731 objects degraded (11.034%) 2024-04-06T03:26:00.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:00 smithi060 ceph-mon[67116]: pgmap v110: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 503 op/s; 198/1743 objects degraded (11.360%) 2024-04-06T03:26:00.462 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:00 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 198/1743 objects degraded (11.360%), 24 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:00.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 219973 10.91 MB/sec execute 364 sec latency 46.975 ms 2024-04-06T03:26:00.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:00 smithi195 ceph-mon[53157]: pgmap v110: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 503 op/s; 198/1743 objects degraded (11.360%) 2024-04-06T03:26:00.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:00 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 198/1743 objects degraded (11.360%), 24 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 220318 10.90 MB/sec execute 365 sec latency 54.080 ms 2024-04-06T03:26:02.529 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:26:02 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:26:02.332+0000 7fbcd10dd7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-06T03:26:02.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 220573 10.88 MB/sec execute 366 sec latency 53.064 ms 2024-04-06T03:26:02.787 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:02 smithi060 ceph-mon[67116]: pgmap v111: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 456 op/s; 198/1743 objects degraded (11.360%) 2024-04-06T03:26:02.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:02 smithi195 ceph-mon[53157]: pgmap v111: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 456 op/s; 198/1743 objects degraded (11.360%) 2024-04-06T03:26:03.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 220951 10.89 MB/sec execute 367 sec latency 36.913 ms 2024-04-06T03:26:03.801 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:26:03 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:26:03.729+0000 7fbcd10dd7c0 -1 osd.1 90 log_to_monitors true 2024-04-06T03:26:04.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 221343 10.90 MB/sec execute 368 sec latency 51.108 ms 2024-04-06T03:26:04.808 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:04 smithi060 ceph-mon[67116]: pgmap v112: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.2 MiB/s wr, 631 op/s; 191/1662 objects degraded (11.492%) 2024-04-06T03:26:04.808 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:04 smithi060 ceph-mon[67116]: from='osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:26:04.809 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:04 smithi060 ceph-mon[67116]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:26:04.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:04 smithi195 ceph-mon[53157]: pgmap v112: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.2 MiB/s wr, 631 op/s; 191/1662 objects degraded (11.492%) 2024-04-06T03:26:04.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:04 smithi195 ceph-mon[53157]: from='osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:26:04.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:04 smithi195 ceph-mon[53157]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:26:05.059 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:26:04 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:26:04.944+0000 7fbcbe52d700 -1 osd.1 90 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:26:05.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 221694 10.89 MB/sec execute 369 sec latency 43.414 ms 2024-04-06T03:26:05.813 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:05 smithi060 ceph-mon[67116]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:26:05.813 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:05 smithi060 ceph-mon[67116]: from='osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:05.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:05 smithi060 ceph-mon[67116]: osdmap e93: 8 total, 7 up, 8 in 2024-04-06T03:26:05.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:05 smithi060 ceph-mon[67116]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:05.814 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:05 smithi060 ceph-mon[67116]: from='osd.1 ' entity='osd.1' 2024-04-06T03:26:05.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:05 smithi195 ceph-mon[53157]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:26:05.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:05 smithi195 ceph-mon[53157]: from='osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:05.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:05 smithi195 ceph-mon[53157]: osdmap e93: 8 total, 7 up, 8 in 2024-04-06T03:26:05.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:05 smithi195 ceph-mon[53157]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:05.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:05 smithi195 ceph-mon[53157]: from='osd.1 ' entity='osd.1' 2024-04-06T03:26:06.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 221791 10.87 MB/sec execute 370 sec latency 748.934 ms 2024-04-06T03:26:06.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: pgmap v114: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.6 MiB/s wr, 547 op/s; 194/1689 objects degraded (11.486%) 2024-04-06T03:26:06.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 191/1662 objects degraded (11.492%), 24 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:06.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:26:06.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949] boot 2024-04-06T03:26:06.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: osdmap e94: 8 total, 8 up, 8 in 2024-04-06T03:26:06.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:26:06.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:06.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:06.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: pgmap v114: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.6 MiB/s wr, 547 op/s; 194/1689 objects degraded (11.486%) 2024-04-06T03:26:06.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 191/1662 objects degraded (11.492%), 24 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:06.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:26:06.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: osd.1 [v2:172.21.15.60:6810/703243949,v1:172.21.15.60:6811/703243949] boot 2024-04-06T03:26:06.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: osdmap e94: 8 total, 8 up, 8 in 2024-04-06T03:26:06.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:26:06.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:06.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:07.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 221791 10.84 MB/sec execute 371 sec latency 1748.961 ms 2024-04-06T03:26:08.181 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:07 smithi060 ceph-mon[67116]: osdmap e95: 8 total, 8 up, 8 in 2024-04-06T03:26:08.181 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:08.182 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:07 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:08.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:07 smithi195 ceph-mon[53157]: osdmap e95: 8 total, 8 up, 8 in 2024-04-06T03:26:08.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:08.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:07 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:08.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 221961 10.82 MB/sec execute 372 sec latency 2107.936 ms 2024-04-06T03:26:09.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:08 smithi060 ceph-mon[67116]: pgmap v117: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 194/1689 objects degraded (11.486%) 2024-04-06T03:26:09.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:08 smithi060 ceph-mon[67116]: osdmap e96: 8 total, 8 up, 8 in 2024-04-06T03:26:09.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:08 smithi195 ceph-mon[53157]: pgmap v117: 97 pgs: 24 active+undersized+degraded, 15 active+undersized, 58 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 194/1689 objects degraded (11.486%) 2024-04-06T03:26:09.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:08 smithi195 ceph-mon[53157]: osdmap e96: 8 total, 8 up, 8 in 2024-04-06T03:26:09.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 222295 10.81 MB/sec execute 373 sec latency 33.819 ms 2024-04-06T03:26:10.215 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:09 smithi060 ceph-mon[67116]: osdmap e97: 8 total, 8 up, 8 in 2024-04-06T03:26:10.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:09 smithi195 ceph-mon[53157]: osdmap e97: 8 total, 8 up, 8 in 2024-04-06T03:26:10.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 222643 10.81 MB/sec execute 374 sec latency 43.748 ms 2024-04-06T03:26:10.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:10 smithi060 ceph-mon[67116]: pgmap v120: 97 pgs: 76 active+clean, 5 active+undersized, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 7 active+undersized+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.7 MiB/s wr, 389 op/s; 176/1563 objects degraded (11.260%); 1.1 MiB/s, 15 objects/s recovering 2024-04-06T03:26:10.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:10 smithi060 ceph-mon[67116]: osdmap e98: 8 total, 8 up, 8 in 2024-04-06T03:26:10.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:10 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 176/1563 objects degraded (11.260%), 15 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:10.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:10 smithi060 ceph-mon[67116]: osdmap e99: 8 total, 8 up, 8 in 2024-04-06T03:26:11.084 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:26:11.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:10 smithi195 ceph-mon[53157]: pgmap v120: 97 pgs: 76 active+clean, 5 active+undersized, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 7 active+undersized+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.7 MiB/s wr, 389 op/s; 176/1563 objects degraded (11.260%); 1.1 MiB/s, 15 objects/s recovering 2024-04-06T03:26:11.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:10 smithi195 ceph-mon[53157]: osdmap e98: 8 total, 8 up, 8 in 2024-04-06T03:26:11.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:10 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 176/1563 objects degraded (11.260%), 15 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:11.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:10 smithi195 ceph-mon[53157]: osdmap e99: 8 total, 8 up, 8 in 2024-04-06T03:26:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 222994 10.81 MB/sec execute 375 sec latency 558.024 ms 2024-04-06T03:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (5m) 5s ago 14m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (101s) 5s ago 14m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (95s) 90s ago 14m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 5s ago 14m - - 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (8m) 5s ago 8m 40.2M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (8m) 90s ago 8m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (4m) 5s ago 15m 505M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (3m) 90s ago 13m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (3m) 5s ago 16m 48.9M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (2m) 90s ago 13m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (8m) 5s ago 8m 279M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (8m) 90s ago 8m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (5m) 5s ago 14m 11.9M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (4m) 90s ago 13m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (62s) 5s ago 13m 248M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (15s) 5s ago 12m 15.2M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (12m) 5s ago 12m 538M 1937M 17.2.0 e1d6a67b021e f09319535abd 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (11m) 5s ago 11m 401M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (11m) 90s ago 11m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (11m) 90s ago 11m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (10m) 90s ago 10m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (10m) 90s ago 10m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (3m) 5s ago 13m 63.0M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:26:12.017 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:11 smithi060 ceph-mon[67116]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:12.017 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:11 smithi060 ceph-mon[67116]: osdmap e100: 8 total, 8 up, 8 in 2024-04-06T03:26:12.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:11 smithi195 ceph-mon[53157]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:12.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:11 smithi195 ceph-mon[53157]: osdmap e100: 8 total, 8 up, 8 in 2024-04-06T03:26:12.614 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:26:12.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:12.616 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:26:12.616 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-06T03:26:12.616 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 6 2024-04-06T03:26:12.616 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:26:12.616 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:26:12.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 223196 10.80 MB/sec execute 376 sec latency 596.287 ms 2024-04-06T03:26:13.052 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "8/23 daemons upgraded", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:26:13.053 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: pgmap v123: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 397 op/s; 145/1560 objects degraded (9.295%); 1.1 MiB/s, 75 keys/s, 21 objects/s recovering 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: pgmap v125: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 862 KiB/s wr, 45 op/s; 145/1560 objects degraded (9.295%); 103 KiB/s, 88 keys/s, 11 objects/s recovering 2024-04-06T03:26:13.559 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: pgmap v126: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 111 KiB/s wr, 10 op/s; 145/1560 objects degraded (9.295%); 31 B/s, 106 keys/s, 8 objects/s recovering 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3778390627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:13 smithi060 ceph-mon[67116]: osdmap e101: 8 total, 8 up, 8 in 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 145/1560 objects degraded (9.295%), 8 pgs degraded 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 145/1560 objects degraded (9.295%), 8 pgs degraded 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.0 is active+recovery_wait+undersized+degraded+remapped, acting [2,6] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.4 is active+recovery_wait+undersized+degraded+remapped, acting [2,5] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.8 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.a is active+recovery_wait+degraded, acting [6,4,1] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.e is active+recovery_wait+undersized+degraded+remapped, acting [7,4] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.18 is active+recovery_wait+degraded, acting [3,0,1] 2024-04-06T03:26:13.576 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1a is active+recovery_wait+undersized+degraded+remapped, acting [4,2] 2024-04-06T03:26:13.577 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1c is active+recovering+degraded, acting [5,4,1] 2024-04-06T03:26:13.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 223390 10.78 MB/sec execute 377 sec latency 756.370 ms 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: pgmap v123: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 397 op/s; 145/1560 objects degraded (9.295%); 1.1 MiB/s, 75 keys/s, 21 objects/s recovering 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:26:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: pgmap v125: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 862 KiB/s wr, 45 op/s; 145/1560 objects degraded (9.295%); 103 KiB/s, 88 keys/s, 11 objects/s recovering 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: pgmap v126: 97 pgs: 1 peering, 87 active+clean, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 111 KiB/s wr, 10 op/s; 145/1560 objects degraded (9.295%); 31 B/s, 106 keys/s, 8 objects/s recovering 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3778390627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:13 smithi195 ceph-mon[53157]: osdmap e101: 8 total, 8 up, 8 in 2024-04-06T03:26:14.583 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:14 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:14.583 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:14 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T03:26:14.583 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:14 smithi060 ceph-mon[67116]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:14.583 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:14 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1138394345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:26:14.583 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:14 smithi060 ceph-mon[67116]: osdmap e102: 8 total, 8 up, 8 in 2024-04-06T03:26:14.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 223676 10.76 MB/sec execute 378 sec latency 73.764 ms 2024-04-06T03:26:14.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:14 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:14.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:14 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-06T03:26:14.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:14 smithi195 ceph-mon[53157]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:14.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:14 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1138394345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:26:14.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:14 smithi195 ceph-mon[53157]: osdmap e102: 8 total, 8 up, 8 in 2024-04-06T03:26:15.625 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:15 smithi060 ceph-mon[67116]: pgmap v129: 97 pgs: 2 peering, 92 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.5 MiB/s wr, 655 op/s; 51/1605 objects degraded (3.178%); 0 B/s, 35 objects/s recovering 2024-04-06T03:26:15.625 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:15 smithi060 ceph-mon[67116]: osdmap e103: 8 total, 8 up, 8 in 2024-04-06T03:26:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 223932 10.75 MB/sec execute 379 sec latency 75.027 ms 2024-04-06T03:26:15.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:15 smithi195 ceph-mon[53157]: pgmap v129: 97 pgs: 2 peering, 92 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.5 MiB/s wr, 655 op/s; 51/1605 objects degraded (3.178%); 0 B/s, 35 objects/s recovering 2024-04-06T03:26:15.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:15 smithi195 ceph-mon[53157]: osdmap e103: 8 total, 8 up, 8 in 2024-04-06T03:26:16.459 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:16 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 51/1605 objects degraded (3.178%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:16.460 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:16 smithi060 ceph-mon[67116]: pgmap v131: 97 pgs: 1 peering, 95 active+clean, 1 active+recovering+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 534 op/s; 3/1605 objects degraded (0.187%); 289 KiB/s, 37 objects/s recovering 2024-04-06T03:26:16.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 224206 10.74 MB/sec execute 380 sec latency 75.881 ms 2024-04-06T03:26:16.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:16 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 51/1605 objects degraded (3.178%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:16.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:16 smithi195 ceph-mon[53157]: pgmap v131: 97 pgs: 1 peering, 95 active+clean, 1 active+recovering+undersized+degraded+remapped; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 534 op/s; 3/1605 objects degraded (0.187%); 289 KiB/s, 37 objects/s recovering 2024-04-06T03:26:17.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 224770 10.76 MB/sec execute 381 sec latency 23.798 ms 2024-04-06T03:26:18.467 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:18 smithi060 ceph-mon[67116]: pgmap v132: 97 pgs: 1 peering, 95 active+clean, 1 active+recovering+undersized+degraded+remapped; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.7 MiB/s wr, 517 op/s; 3/1662 objects degraded (0.181%); 193 KiB/s, 24 objects/s recovering 2024-04-06T03:26:18.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 225255 10.77 MB/sec execute 382 sec latency 40.908 ms 2024-04-06T03:26:18.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:18 smithi195 ceph-mon[53157]: pgmap v132: 97 pgs: 1 peering, 95 active+clean, 1 active+recovering+undersized+degraded+remapped; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.7 MiB/s wr, 517 op/s; 3/1662 objects degraded (0.181%); 193 KiB/s, 24 objects/s recovering 2024-04-06T03:26:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 225846 10.78 MB/sec execute 383 sec latency 27.453 ms 2024-04-06T03:26:20.481 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:20 smithi060 ceph-mon[67116]: pgmap v133: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.3 MiB/s wr, 630 op/s; 149 KiB/s, 19 objects/s recovering 2024-04-06T03:26:20.481 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:20 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/1662 objects degraded (0.181%), 1 pg degraded) 2024-04-06T03:26:20.657 INFO:teuthology.orchestra.run.smithi060.stdout: 5 226395 10.80 MB/sec execute 384 sec latency 40.933 ms 2024-04-06T03:26:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:20 smithi195 ceph-mon[53157]: pgmap v133: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.3 MiB/s wr, 630 op/s; 149 KiB/s, 19 objects/s recovering 2024-04-06T03:26:20.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:20 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/1662 objects degraded (0.181%), 1 pg degraded) 2024-04-06T03:26:21.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 226841 10.80 MB/sec execute 385 sec latency 56.731 ms 2024-04-06T03:26:22.520 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:22 smithi060 ceph-mon[67116]: pgmap v134: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.8 MiB/s wr, 345 op/s; 132 KiB/s, 5 objects/s recovering 2024-04-06T03:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 227230 10.78 MB/sec execute 386 sec latency 59.881 ms 2024-04-06T03:26:22.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:22 smithi195 ceph-mon[53157]: pgmap v134: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.8 MiB/s wr, 345 op/s; 132 KiB/s, 5 objects/s recovering 2024-04-06T03:26:23.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 227551 10.78 MB/sec execute 387 sec latency 62.996 ms 2024-04-06T03:26:24.590 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:24 smithi060 ceph-mon[67116]: pgmap v135: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 498 op/s; 116 KiB/s, 5 objects/s recovering 2024-04-06T03:26:24.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 227790 10.77 MB/sec execute 388 sec latency 75.408 ms 2024-04-06T03:26:24.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:24 smithi195 ceph-mon[53157]: pgmap v135: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 498 op/s; 116 KiB/s, 5 objects/s recovering 2024-04-06T03:26:25.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 228065 10.78 MB/sec execute 389 sec latency 63.592 ms 2024-04-06T03:26:26.474 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:26 smithi060 ceph-mon[67116]: pgmap v136: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.2 MiB/s wr, 517 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:26.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 228322 10.76 MB/sec execute 390 sec latency 130.806 ms 2024-04-06T03:26:26.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:26 smithi195 ceph-mon[53157]: pgmap v136: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.2 MiB/s wr, 517 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:27.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 228631 10.76 MB/sec execute 391 sec latency 53.532 ms 2024-04-06T03:26:28.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:28.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:28.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:28.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:28.534 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:28.534 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:28.534 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:28.534 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:28.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 228883 10.75 MB/sec execute 392 sec latency 100.667 ms 2024-04-06T03:26:29.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:29 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:29.546 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:29 smithi060 ceph-mon[67116]: Upgrade: osd.2 is safe to restart 2024-04-06T03:26:29.546 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:29 smithi060 ceph-mon[67116]: pgmap v137: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.7 MiB/s wr, 616 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:29.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 229196 10.74 MB/sec execute 393 sec latency 41.360 ms 2024-04-06T03:26:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:29 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T03:26:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:29 smithi195 ceph-mon[53157]: Upgrade: osd.2 is safe to restart 2024-04-06T03:26:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:29 smithi195 ceph-mon[53157]: pgmap v137: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.7 MiB/s wr, 616 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:30.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 229517 10.73 MB/sec execute 394 sec latency 45.964 ms 2024-04-06T03:26:31.322 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: Upgrade: Updating osd.2 2024-04-06T03:26:31.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:31.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:26:31.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:31.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: Deploying daemon osd.2 on smithi060 2024-04-06T03:26:31.323 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:31 smithi060 ceph-mon[67116]: pgmap v138: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.2 MiB/s wr, 592 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:31.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: Upgrade: Updating osd.2 2024-04-06T03:26:31.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:31.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:26:31.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:31.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: Deploying daemon osd.2 on smithi060 2024-04-06T03:26:31.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:31 smithi195 ceph-mon[53157]: pgmap v138: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.2 MiB/s wr, 592 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T03:26:31.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 229781 10.73 MB/sec execute 395 sec latency 64.569 ms 2024-04-06T03:26:32.609 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:32 smithi060 ceph-mon[67116]: pgmap v139: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 MiB/s wr, 443 op/s 2024-04-06T03:26:32.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 230029 10.71 MB/sec execute 396 sec latency 81.236 ms 2024-04-06T03:26:32.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:32 smithi195 ceph-mon[53157]: pgmap v139: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 MiB/s wr, 443 op/s 2024-04-06T03:26:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 230319 10.71 MB/sec execute 397 sec latency 64.379 ms 2024-04-06T03:26:34.450 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:34 smithi060 ceph-mon[67116]: pgmap v140: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 602 op/s 2024-04-06T03:26:34.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 230559 10.69 MB/sec execute 398 sec latency 71.921 ms 2024-04-06T03:26:34.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:34 smithi195 ceph-mon[53157]: pgmap v140: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 602 op/s 2024-04-06T03:26:35.224 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:35 smithi060 systemd[1]: Stopping Ceph osd.2 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:26:35.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 230839 10.67 MB/sec execute 399 sec latency 97.987 ms 2024-04-06T03:26:35.727 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:35 smithi060 ceph-mon[67116]: osd.2 marked itself down and dead 2024-04-06T03:26:35.727 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:35 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[42063]: 2024-04-06T03:26:35.492+0000 7f06bbdba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:26:35.727 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:35 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[42063]: 2024-04-06T03:26:35.492+0000 7f06bbdba700 -1 osd.2 103 *** Got signal Terminated *** 2024-04-06T03:26:35.727 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:35 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[42063]: 2024-04-06T03:26:35.492+0000 7f06bbdba700 -1 osd.2 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:26:35.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:35 smithi195 ceph-mon[53157]: osd.2 marked itself down and dead 2024-04-06T03:26:36.248 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:36 smithi060 podman[78230]: 2024-04-06 03:26:36.204471882 +0000 UTC m=+0.848011627 container died f09319535abd0efb9b9dd23bb80cef2601c6981df1c4baa6399eea2d2fc2ecdd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, name=centos-stream) 2024-04-06T03:26:36.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 230935 10.65 MB/sec execute 400 sec latency 1015.824 ms 2024-04-06T03:26:36.730 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:36 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:26:36.731 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:36 smithi060 ceph-mon[67116]: osdmap e104: 8 total, 7 up, 8 in 2024-04-06T03:26:36.731 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:36 smithi060 ceph-mon[67116]: pgmap v142: 97 pgs: 5 stale+active+clean, 92 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 MiB/s wr, 415 op/s 2024-04-06T03:26:36.731 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:36 smithi060 podman[78230]: 2024-04-06 03:26:36.567402954 +0000 UTC m=+1.210942698 container cleanup f09319535abd0efb9b9dd23bb80cef2601c6981df1c4baa6399eea2d2fc2ecdd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-04-06T03:26:36.731 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:36 smithi060 bash[78230]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T03:26:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:36 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:26:36.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:36 smithi195 ceph-mon[53157]: osdmap e104: 8 total, 7 up, 8 in 2024-04-06T03:26:36.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:36 smithi195 ceph-mon[53157]: pgmap v142: 97 pgs: 5 stale+active+clean, 92 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 MiB/s wr, 415 op/s 2024-04-06T03:26:37.234 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:37 smithi060 podman[78246]: 2024-04-06 03:26:37.069953267 +0000 UTC m=+0.863091607 container remove f09319535abd0efb9b9dd23bb80cef2601c6981df1c4baa6399eea2d2fc2ecdd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-type=git, ceph=True) 2024-04-06T03:26:37.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 231309 10.65 MB/sec execute 401 sec latency 45.808 ms 2024-04-06T03:26:37.736 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:37 smithi060 ceph-mon[67116]: osdmap e105: 8 total, 7 up, 8 in 2024-04-06T03:26:37.737 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:37 smithi060 podman[78333]: 2024-04-06 03:26:37.41222057 +0000 UTC m=+0.037713060 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:26:37.737 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:37 smithi060 podman[78333]: 2024-04-06 03:26:37.557851597 +0000 UTC m=+0.183344086 container create 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:26:37.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:37 smithi195 ceph-mon[53157]: osdmap e105: 8 total, 7 up, 8 in 2024-04-06T03:26:37.987 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:37 smithi060 podman[78333]: 2024-04-06 03:26:37.92691168 +0000 UTC m=+0.552404170 container init 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:26:37.988 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:37 smithi060 podman[78333]: 2024-04-06 03:26:37.935019191 +0000 UTC m=+0.560511673 container start 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:26:38.240 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:38 smithi060 podman[78333]: 2024-04-06 03:26:38.093435655 +0000 UTC m=+0.718928144 container attach 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235) 2024-04-06T03:26:38.497 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:38 smithi060 podman[78333]: 2024-04-06 03:26:38.413650052 +0000 UTC m=+1.039142539 container died 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-06T03:26:38.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 231824 10.68 MB/sec execute 402 sec latency 45.024 ms 2024-04-06T03:26:38.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:38 smithi060 ceph-mon[67116]: pgmap v144: 97 pgs: 2 active+undersized, 9 active+undersized+degraded, 4 stale+active+clean, 82 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.3 MiB/s wr, 301 op/s; 66/1650 objects degraded (4.000%) 2024-04-06T03:26:38.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:38 smithi195 ceph-mon[53157]: pgmap v144: 97 pgs: 2 active+undersized, 9 active+undersized+degraded, 4 stale+active+clean, 82 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.3 MiB/s wr, 301 op/s; 66/1650 objects degraded (4.000%) 2024-04-06T03:26:39.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 232337 10.69 MB/sec execute 403 sec latency 31.706 ms 2024-04-06T03:26:39.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:39 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 66/1650 objects degraded (4.000%), 9 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:39.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:39 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 66/1650 objects degraded (4.000%), 9 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:40.006 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:39 smithi060 podman[78359]: 2024-04-06 03:26:39.793591485 +0000 UTC m=+1.370053661 container remove 1dc6203719cb9a9a44419b82c0a965bca3a2c43a84ac0afbb3c76b5a7e71d25f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:26:40.006 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:39 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-04-06T03:26:40.006 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:39 smithi060 systemd[1]: Stopped Ceph osd.2 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:26:40.006 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:39 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.2.service: Consumed 3min 860ms CPU time. 2024-04-06T03:26:40.258 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 systemd[1]: Starting Ceph osd.2 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:26:40.509 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 podman[78446]: 2024-04-06 03:26:40.328476053 +0000 UTC m=+0.026216825 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:26:40.509 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 podman[78446]: 2024-04-06 03:26:40.461776116 +0000 UTC m=+0.159516913 container create 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-06T03:26:40.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 232552 10.67 MB/sec execute 404 sec latency 32.876 ms 2024-04-06T03:26:40.759 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:40 smithi060 ceph-mon[67116]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 606 op/s; 161/1698 objects degraded (9.482%) 2024-04-06T03:26:40.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:40 smithi195 ceph-mon[53157]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 606 op/s; 161/1698 objects degraded (9.482%) 2024-04-06T03:26:41.011 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 podman[78446]: 2024-04-06 03:26:40.765277319 +0000 UTC m=+0.463018080 container init 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235) 2024-04-06T03:26:41.011 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 podman[78446]: 2024-04-06 03:26:40.770390257 +0000 UTC m=+0.468131012 container start 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-06T03:26:41.011 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:40 smithi060 podman[78446]: 2024-04-06 03:26:40.950845584 +0000 UTC m=+0.648586338 container attach 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-06T03:26:41.516 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T03:26:41.516 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T03:26:41.516 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T03:26:41.516 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T03:26:41.516 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate[78460]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 bash[78446]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 conmon[78460]: conmon 7c41035a8e7a2b34259b : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3.scope/container/memory.events 2024-04-06T03:26:41.517 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:41 smithi060 podman[78446]: 2024-04-06 03:26:41.487441107 +0000 UTC m=+1.185181865 container died 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-06T03:26:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 232870 10.67 MB/sec execute 405 sec latency 38.610 ms 2024-04-06T03:26:42.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 233218 10.66 MB/sec execute 406 sec latency 37.631 ms 2024-04-06T03:26:42.791 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:42 smithi060 podman[78581]: 2024-04-06 03:26:42.57303551 +0000 UTC m=+1.078902681 container remove 7c41035a8e7a2b34259b29f1be9a3cb0688cf9091e4118dac1720e8d3115e8a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-activate, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-06T03:26:43.042 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:42 smithi060 podman[78639]: 2024-04-06 03:26:42.718380306 +0000 UTC m=+0.024708138 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:26:43.043 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:42 smithi060 podman[78639]: 2024-04-06 03:26:42.851984254 +0000 UTC m=+0.158312068 container create 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-06T03:26:43.552 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:43.553 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:43.553 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:43 smithi060 ceph-mon[67116]: pgmap v146: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 367 op/s; 161/1698 objects degraded (9.482%) 2024-04-06T03:26:43.553 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:43 smithi060 podman[78639]: 2024-04-06 03:26:43.50726668 +0000 UTC m=+0.813594494 container init 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-06T03:26:43.553 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:43 smithi060 podman[78639]: 2024-04-06 03:26:43.518167103 +0000 UTC m=+0.824494911 container start 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:26:43.553 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:43 smithi060 ceph-osd[78658]: -- 172.21.15.60:0/4076440827 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555f1254d680 con 0x555f12516c00 2024-04-06T03:26:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 233552 10.67 MB/sec execute 407 sec latency 47.324 ms 2024-04-06T03:26:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:43.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:43 smithi195 ceph-mon[53157]: pgmap v146: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 367 op/s; 161/1698 objects degraded (9.482%) 2024-04-06T03:26:43.809 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:43 smithi060 bash[78639]: 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 2024-04-06T03:26:43.810 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:43 smithi060 systemd[1]: Started Ceph osd.2 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:26:44.178 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:26:44.327 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:26:44.085+0000 7f192a3667c0 -1 Falling back to public interface 2024-04-06T03:26:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 233887 10.66 MB/sec execute 408 sec latency 53.952 ms 2024-04-06T03:26:44.846 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:44.846 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:44.846 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:44.847 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-mon[67116]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:44.847 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:44 smithi060 ceph-mon[67116]: pgmap v147: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 429 op/s; 165/1737 objects degraded (9.499%) 2024-04-06T03:26:45.095 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:26:45.095 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (6m) 39s ago 15m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (2m) 39s ago 15m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (2m) 2m ago 14m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 39s ago 14m - - 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (9m) 39s ago 9m 40.2M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (9m) 2m ago 9m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (4m) 39s ago 16m 505M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (3m) 39s ago 16m 48.9M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (3m) 2m ago 14m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (9m) 39s ago 9m 279M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:26:45.096 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (8m) 2m ago 8m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (5m) 39s ago 14m 11.9M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (5m) 2m ago 14m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (95s) 39s ago 13m 248M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (48s) 39s ago 13m 15.2M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 starting - - - 1937M 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (12m) 39s ago 12m 401M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (12m) 2m ago 12m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (11m) 2m ago 11m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (11m) 2m ago 11m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (10m) 2m ago 10m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:26:45.097 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (4m) 39s ago 14m 63.0M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:26:45.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:45.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:45.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:44 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:45.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:44 smithi195 ceph-mon[53157]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:45.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:44 smithi195 ceph-mon[53157]: pgmap v147: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 429 op/s; 165/1737 objects degraded (9.499%) 2024-04-06T03:26:45.658 INFO:teuthology.orchestra.run.smithi060.stdout: 5 234244 10.64 MB/sec execute 409 sec latency 50.586 ms 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:26:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 6 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:26:45.702 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:26:45.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:45 smithi060 ceph-mon[67116]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:45.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:45 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 165/1737 objects degraded (9.499%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:45.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:45 smithi060 ceph-mon[67116]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:45.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:45 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3784280870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:46.122 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:45 smithi195 ceph-mon[53157]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:46.123 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:45 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 165/1737 objects degraded (9.499%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:46.123 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:45 smithi195 ceph-mon[53157]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:46.123 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:45 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3784280870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:26:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "8/23 daemons upgraded", 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:26:46.201 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:26:46.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 234616 10.64 MB/sec execute 410 sec latency 39.733 ms 2024-04-06T03:26:46.826 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 165/1719 objects degraded (9.599%), 19 pgs degraded 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: osd.2 (root=default,host=smithi060) is down 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 165/1719 objects degraded (9.599%), 19 pgs degraded 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-04-06T03:26:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-04-06T03:26:46.828 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-04-06T03:26:46.871 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:46 smithi060 ceph-mon[67116]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:46.872 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:46 smithi060 ceph-mon[67116]: pgmap v148: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 483 op/s; 165/1719 objects degraded (9.599%) 2024-04-06T03:26:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:46 smithi195 ceph-mon[53157]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:26:47.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:46 smithi195 ceph-mon[53157]: pgmap v148: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 483 op/s; 165/1719 objects degraded (9.599%) 2024-04-06T03:26:47.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 234869 10.63 MB/sec execute 411 sec latency 60.698 ms 2024-04-06T03:26:47.886 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:47 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/350642524' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:26:48.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:47 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/350642524' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:26:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 235295 10.65 MB/sec execute 412 sec latency 60.317 ms 2024-04-06T03:26:48.896 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:48 smithi060 ceph-mon[67116]: pgmap v149: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 159/1692 objects degraded (9.397%) 2024-04-06T03:26:49.221 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:48 smithi195 ceph-mon[53157]: pgmap v149: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 486 op/s; 159/1692 objects degraded (9.397%) 2024-04-06T03:26:49.645 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:49 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:26:49.505+0000 7f192a3667c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-06T03:26:49.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 235894 10.66 MB/sec execute 413 sec latency 31.741 ms 2024-04-06T03:26:50.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 236270 10.66 MB/sec execute 414 sec latency 32.422 ms 2024-04-06T03:26:50.659 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:50 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 155/1644 objects degraded (9.428%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:50.911 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:50 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:26:50.668+0000 7f192a3667c0 -1 osd.2 103 log_to_monitors true 2024-04-06T03:26:50.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:50 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 155/1644 objects degraded (9.428%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 236713 10.66 MB/sec execute 415 sec latency 38.630 ms 2024-04-06T03:26:51.670 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:51 smithi060 ceph-mon[67116]: pgmap v150: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 602 op/s; 155/1644 objects degraded (9.428%) 2024-04-06T03:26:51.670 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:51 smithi060 ceph-mon[67116]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:26:51.671 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:26:51 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:26:51.547+0000 7f191b7be700 -1 osd.2 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:26:51.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:51 smithi195 ceph-mon[53157]: pgmap v150: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 602 op/s; 155/1644 objects degraded (9.428%) 2024-04-06T03:26:51.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:51 smithi195 ceph-mon[53157]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:26:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237066 10.66 MB/sec execute 416 sec latency 165.382 ms 2024-04-06T03:26:52.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:52 smithi060 ceph-mon[67116]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:26:52.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:52 smithi060 ceph-mon[67116]: osdmap e106: 8 total, 7 up, 8 in 2024-04-06T03:26:52.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:52 smithi060 ceph-mon[67116]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:52.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:52 smithi195 ceph-mon[53157]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:26:52.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:52 smithi195 ceph-mon[53157]: osdmap e106: 8 total, 7 up, 8 in 2024-04-06T03:26:52.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:52 smithi195 ceph-mon[53157]: from='osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:26:53.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237066 10.64 MB/sec execute 417 sec latency 1165.413 ms 2024-04-06T03:26:53.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: pgmap v152: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 478 op/s; 155/1644 objects degraded (9.428%) 2024-04-06T03:26:53.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661] boot 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: osdmap e107: 8 total, 8 up, 8 in 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:53 smithi195 ceph-mon[53157]: osdmap e108: 8 total, 8 up, 8 in 2024-04-06T03:26:53.997 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: pgmap v152: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 478 op/s; 155/1644 objects degraded (9.428%) 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: osd.2 [v2:172.21.15.60:6818/440417661,v1:172.21.15.60:6819/440417661] boot 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: osdmap e107: 8 total, 8 up, 8 in 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:53.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:53 smithi060 ceph-mon[67116]: osdmap e108: 8 total, 8 up, 8 in 2024-04-06T03:26:54.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237066 10.61 MB/sec execute 418 sec latency 2158.195 ms 2024-04-06T03:26:55.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:55 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:55.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:55 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:55.175 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:55 smithi060 ceph-mon[67116]: osdmap e109: 8 total, 8 up, 8 in 2024-04-06T03:26:55.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:55 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:55.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:55 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:55.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:55 smithi195 ceph-mon[53157]: osdmap e109: 8 total, 8 up, 8 in 2024-04-06T03:26:55.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237343 10.60 MB/sec execute 419 sec latency 2158.205 ms 2024-04-06T03:26:56.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:56 smithi195 ceph-mon[53157]: pgmap v155: 97 pgs: 15 peering, 4 active+undersized, 7 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 579 op/s; 67/1695 objects degraded (3.953%) 2024-04-06T03:26:56.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:56 smithi195 ceph-mon[53157]: osdmap e110: 8 total, 8 up, 8 in 2024-04-06T03:26:56.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:56 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 67/1695 objects degraded (3.953%), 7 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:56.497 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:56 smithi060 ceph-mon[67116]: pgmap v155: 97 pgs: 15 peering, 4 active+undersized, 7 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 579 op/s; 67/1695 objects degraded (3.953%) 2024-04-06T03:26:56.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:56 smithi060 ceph-mon[67116]: osdmap e110: 8 total, 8 up, 8 in 2024-04-06T03:26:56.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:56 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 67/1695 objects degraded (3.953%), 7 pgs degraded (PG_DEGRADED) 2024-04-06T03:26:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237360 10.58 MB/sec execute 420 sec latency 1038.508 ms 2024-04-06T03:26:57.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:57 smithi060 ceph-mon[67116]: pgmap v158: 97 pgs: 1 remapped+peering, 15 peering, 1 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 74 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 5.9 MiB/s wr, 496 op/s; 77/1674 objects degraded (4.600%) 2024-04-06T03:26:57.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:57 smithi060 ceph-mon[67116]: osdmap e111: 8 total, 8 up, 8 in 2024-04-06T03:26:57.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:57.545 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:57.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 237680 10.57 MB/sec execute 421 sec latency 976.346 ms 2024-04-06T03:26:57.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:57 smithi195 ceph-mon[53157]: pgmap v158: 97 pgs: 1 remapped+peering, 15 peering, 1 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 74 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 5.9 MiB/s wr, 496 op/s; 77/1674 objects degraded (4.600%) 2024-04-06T03:26:57.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:57 smithi195 ceph-mon[53157]: osdmap e111: 8 total, 8 up, 8 in 2024-04-06T03:26:57.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:26:57.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:58.549 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:58 smithi060 ceph-mon[67116]: osdmap e112: 8 total, 8 up, 8 in 2024-04-06T03:26:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 238097 10.56 MB/sec execute 422 sec latency 860.229 ms 2024-04-06T03:26:58.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:58 smithi195 ceph-mon[53157]: osdmap e112: 8 total, 8 up, 8 in 2024-04-06T03:26:59.555 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:59 smithi060 ceph-mon[67116]: pgmap v161: 97 pgs: 1 activating, 1 remapped+peering, 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.0 MiB/s wr, 326 op/s; 57/1662 objects degraded (3.430%); 1.6 MiB/s, 18 objects/s recovering 2024-04-06T03:26:59.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:26:59 smithi060 ceph-mon[67116]: osdmap e113: 8 total, 8 up, 8 in 2024-04-06T03:26:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 238451 10.55 MB/sec execute 423 sec latency 54.518 ms 2024-04-06T03:26:59.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:59 smithi195 ceph-mon[53157]: pgmap v161: 97 pgs: 1 activating, 1 remapped+peering, 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.0 MiB/s wr, 326 op/s; 57/1662 objects degraded (3.430%); 1.6 MiB/s, 18 objects/s recovering 2024-04-06T03:26:59.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:26:59 smithi195 ceph-mon[53157]: osdmap e113: 8 total, 8 up, 8 in 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: pgmap v163: 97 pgs: 1 activating, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 2.8 MiB/s wr, 345 op/s; 57/1638 objects degraded (3.480%); 2.6 MiB/s, 35 objects/s recovering 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.568 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:00.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:00.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:00 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-06T03:27:00.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 238908 10.57 MB/sec execute 424 sec latency 55.113 ms 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: pgmap v163: 97 pgs: 1 activating, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 2.8 MiB/s wr, 345 op/s; 57/1638 objects degraded (3.480%); 2.6 MiB/s, 35 objects/s recovering 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:00.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:00.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:00.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:00 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-06T03:27:01.608 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:01 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 57/1638 objects degraded (3.480%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 239252 10.56 MB/sec execute 425 sec latency 42.963 ms 2024-04-06T03:27:01.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:01 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 57/1638 objects degraded (3.480%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:02.629 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:02 smithi060 ceph-mon[67116]: pgmap v164: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 3.4 MiB/s wr, 496 op/s; 2.2 MiB/s, 38 objects/s recovering 2024-04-06T03:27:02.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 239601 10.57 MB/sec execute 426 sec latency 42.004 ms 2024-04-06T03:27:02.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:02 smithi195 ceph-mon[53157]: pgmap v164: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 3.4 MiB/s wr, 496 op/s; 2.2 MiB/s, 38 objects/s recovering 2024-04-06T03:27:03.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:03 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/1638 objects degraded (3.480%), 2 pgs degraded) 2024-04-06T03:27:03.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 239906 10.57 MB/sec execute 427 sec latency 40.273 ms 2024-04-06T03:27:03.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:03 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/1638 objects degraded (3.480%), 2 pgs degraded) 2024-04-06T03:27:04.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 240189 10.56 MB/sec execute 428 sec latency 44.541 ms 2024-04-06T03:27:04.684 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:04 smithi060 ceph-mon[67116]: pgmap v165: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 610 op/s; 1.6 MiB/s, 28 objects/s recovering 2024-04-06T03:27:04.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:04 smithi195 ceph-mon[53157]: pgmap v165: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 610 op/s; 1.6 MiB/s, 28 objects/s recovering 2024-04-06T03:27:05.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 240458 10.55 MB/sec execute 429 sec latency 53.663 ms 2024-04-06T03:27:06.659 INFO:teuthology.orchestra.run.smithi060.stdout: 5 240699 10.54 MB/sec execute 430 sec latency 55.821 ms 2024-04-06T03:27:06.729 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:06 smithi060 ceph-mon[67116]: pgmap v166: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.1 MiB/s wr, 507 op/s; 1.1 MiB/s, 24 objects/s recovering 2024-04-06T03:27:06.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:06 smithi195 ceph-mon[53157]: pgmap v166: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.1 MiB/s wr, 507 op/s; 1.1 MiB/s, 24 objects/s recovering 2024-04-06T03:27:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 240948 10.52 MB/sec execute 431 sec latency 59.109 ms 2024-04-06T03:27:08.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 241193 10.52 MB/sec execute 432 sec latency 63.328 ms 2024-04-06T03:27:08.805 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:08 smithi060 ceph-mon[67116]: pgmap v167: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.3 MiB/s wr, 430 op/s; 538 KiB/s, 13 objects/s recovering 2024-04-06T03:27:08.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:08 smithi195 ceph-mon[53157]: pgmap v167: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.3 MiB/s wr, 430 op/s; 538 KiB/s, 13 objects/s recovering 2024-04-06T03:27:09.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 241391 10.50 MB/sec execute 433 sec latency 97.830 ms 2024-04-06T03:27:10.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 241719 10.49 MB/sec execute 434 sec latency 46.739 ms 2024-04-06T03:27:10.663 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:10 smithi060 ceph-mon[67116]: pgmap v168: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.5 MiB/s wr, 430 op/s; 75 KiB/s, 7 objects/s recovering 2024-04-06T03:27:10.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:10 smithi195 ceph-mon[53157]: pgmap v168: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.5 MiB/s wr, 430 op/s; 75 KiB/s, 7 objects/s recovering 2024-04-06T03:27:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 241984 10.48 MB/sec execute 435 sec latency 57.923 ms 2024-04-06T03:27:12.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 242366 10.49 MB/sec execute 436 sec latency 50.508 ms 2024-04-06T03:27:13.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:13 smithi060 ceph-mon[67116]: pgmap v169: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 448 op/s; 57 KiB/s, 4 objects/s recovering 2024-04-06T03:27:13.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:13.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:13.493 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:13.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 242692 10.49 MB/sec execute 437 sec latency 67.279 ms 2024-04-06T03:27:13.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:13 smithi195 ceph-mon[53157]: pgmap v169: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 448 op/s; 57 KiB/s, 4 objects/s recovering 2024-04-06T03:27:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:13.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:14.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 243030 10.49 MB/sec execute 438 sec latency 53.676 ms 2024-04-06T03:27:15.511 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:15 smithi060 ceph-mon[67116]: pgmap v170: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 457 op/s 2024-04-06T03:27:15.512 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 243327 10.49 MB/sec execute 439 sec latency 59.024 ms 2024-04-06T03:27:15.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:15 smithi195 ceph-mon[53157]: pgmap v170: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 457 op/s 2024-04-06T03:27:15.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:16.528 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:16 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:16.528 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:16 smithi060 ceph-mon[67116]: Upgrade: osd.3 is safe to restart 2024-04-06T03:27:16.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 243692 10.49 MB/sec execute 440 sec latency 38.959 ms 2024-04-06T03:27:16.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:16 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T03:27:16.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:16 smithi195 ceph-mon[53157]: Upgrade: osd.3 is safe to restart 2024-04-06T03:27:17.471 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:27:17.544 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:17 smithi060 ceph-mon[67116]: pgmap v171: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 415 op/s 2024-04-06T03:27:17.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 244078 10.49 MB/sec execute 441 sec latency 45.055 ms 2024-04-06T03:27:17.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:17 smithi195 ceph-mon[53157]: pgmap v171: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 415 op/s 2024-04-06T03:27:18.526 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (6m) 25s ago 15m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (2m) 25s ago 15m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (2m) 2m ago 15m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 25s ago 15m - - 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (9m) 25s ago 9m 40.5M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (9m) 2m ago 9m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (5m) 25s ago 17m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (4m) 2m ago 15m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (4m) 25s ago 17m 49.9M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (3m) 2m ago 15m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (9m) 25s ago 9m 281M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:27:18.527 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (9m) 2m ago 9m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (6m) 25s ago 15m 11.4M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (5m) 2m ago 14m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (2m) 25s ago 14m 305M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (82s) 25s ago 13m 257M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (35s) 25s ago 13m 12.7M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (12m) 25s ago 12m 424M 1937M 17.2.0 e1d6a67b021e fb37621b36d2 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (12m) 2m ago 12m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (12m) 2m ago 12m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (11m) 2m ago 11m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (11m) 2m ago 11m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:27:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (4m) 25s ago 14m 63.1M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:27:18.555 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: from='client.44235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:18.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: Upgrade: Updating osd.3 2024-04-06T03:27:18.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:18.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:27:18.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:18.556 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: Deploying daemon osd.3 on smithi060 2024-04-06T03:27:18.557 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: pgmap v172: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 355 op/s 2024-04-06T03:27:18.557 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:18 smithi060 ceph-mon[67116]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:18.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 244319 10.47 MB/sec execute 442 sec latency 53.840 ms 2024-04-06T03:27:18.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: from='client.44235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: Upgrade: Updating osd.3 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: Deploying daemon osd.3 on smithi060 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: pgmap v172: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 355 op/s 2024-04-06T03:27:18.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:18 smithi195 ceph-mon[53157]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:19.174 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:27:19.174 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:19.175 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:27:19.176 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-06T03:27:19.176 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 7 2024-04-06T03:27:19.176 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:27:19.176 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:27:19.565 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:19 smithi060 ceph-mon[67116]: from='client.44243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:19.565 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:19 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/209907781' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 244612 10.46 MB/sec execute 443 sec latency 56.718 ms 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:27:19.723 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "9/23 daemons upgraded", 2024-04-06T03:27:19.724 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:27:19.724 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:27:19.724 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:27:19.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:19 smithi195 ceph-mon[53157]: from='client.44243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:19.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:19 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/209907781' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:20.362 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:27:20.362 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:27:20.363 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:27:20.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:20 smithi060 ceph-mon[67116]: pgmap v173: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 495 op/s 2024-04-06T03:27:20.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:20 smithi060 ceph-mon[67116]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:20.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:20 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1393626075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:27:20.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 244914 10.45 MB/sec execute 444 sec latency 60.930 ms 2024-04-06T03:27:20.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:20 smithi195 ceph-mon[53157]: pgmap v173: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 495 op/s 2024-04-06T03:27:20.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:20 smithi195 ceph-mon[53157]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:20.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:20 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1393626075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:27:21.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 245296 10.45 MB/sec execute 445 sec latency 45.188 ms 2024-04-06T03:27:22.622 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:22 smithi060 ceph-mon[67116]: pgmap v174: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.7 MiB/s wr, 522 op/s 2024-04-06T03:27:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 245579 10.44 MB/sec execute 446 sec latency 48.219 ms 2024-04-06T03:27:22.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:22 smithi195 ceph-mon[53157]: pgmap v174: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.7 MiB/s wr, 522 op/s 2024-04-06T03:27:23.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 246006 10.45 MB/sec execute 447 sec latency 39.122 ms 2024-04-06T03:27:23.904 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:23 smithi060 systemd[1]: Stopping Ceph osd.3 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:27:24.175 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:24 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[45896]: 2024-04-06T03:27:24.090+0000 7ffac2f91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:27:24.175 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:24 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[45896]: 2024-04-06T03:27:24.090+0000 7ffac2f91700 -1 osd.3 113 *** Got signal Terminated *** 2024-04-06T03:27:24.175 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:24 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[45896]: 2024-04-06T03:27:24.090+0000 7ffac2f91700 -1 osd.3 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:27:24.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 246321 10.45 MB/sec execute 448 sec latency 62.126 ms 2024-04-06T03:27:24.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:24 smithi060 ceph-mon[67116]: pgmap v175: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.1 MiB/s wr, 510 op/s 2024-04-06T03:27:24.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:24 smithi060 ceph-mon[67116]: osd.3 marked itself down and dead 2024-04-06T03:27:24.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:24 smithi195 ceph-mon[53157]: pgmap v175: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.1 MiB/s wr, 510 op/s 2024-04-06T03:27:24.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:24 smithi195 ceph-mon[53157]: osd.3 marked itself down and dead 2024-04-06T03:27:25.578 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:25 smithi060 podman[80856]: 2024-04-06 03:27:25.290451126 +0000 UTC m=+1.391637173 container died fb37621b36d211dd43660b8df09914ef7e1c48e1ac4c71f75212f773a79bda37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, release=754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T03:27:25.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 246322 10.42 MB/sec execute 449 sec latency 1050.141 ms 2024-04-06T03:27:25.829 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:25 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:27:25.829 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:25 smithi060 ceph-mon[67116]: osdmap e114: 8 total, 7 up, 8 in 2024-04-06T03:27:25.829 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:25 smithi060 podman[80856]: 2024-04-06 03:27:25.787415274 +0000 UTC m=+1.888601325 container cleanup fb37621b36d211dd43660b8df09914ef7e1c48e1ac4c71f75212f773a79bda37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container) 2024-04-06T03:27:25.829 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:25 smithi060 bash[80856]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T03:27:25.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:25 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:27:25.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:25 smithi195 ceph-mon[53157]: osdmap e114: 8 total, 7 up, 8 in 2024-04-06T03:27:26.352 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:26 smithi060 podman[80873]: 2024-04-06 03:27:26.203675115 +0000 UTC m=+0.906726269 container remove fb37621b36d211dd43660b8df09914ef7e1c48e1ac4c71f75212f773a79bda37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, release=754, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc.) 2024-04-06T03:27:26.606 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:26 smithi060 podman[80960]: 2024-04-06 03:27:26.504393404 +0000 UTC m=+0.021742691 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:27:26.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 246684 10.43 MB/sec execute 450 sec latency 1040.725 ms 2024-04-06T03:27:26.858 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:26 smithi060 ceph-mon[67116]: pgmap v177: 97 pgs: 10 peering, 12 stale+active+clean, 75 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-04-06T03:27:26.858 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:26 smithi060 ceph-mon[67116]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-06T03:27:26.858 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:26 smithi060 ceph-mon[67116]: osdmap e115: 8 total, 7 up, 8 in 2024-04-06T03:27:26.859 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:26 smithi060 podman[80960]: 2024-04-06 03:27:26.630200291 +0000 UTC m=+0.147549557 container create 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-06T03:27:26.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:26 smithi195 ceph-mon[53157]: pgmap v177: 97 pgs: 10 peering, 12 stale+active+clean, 75 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-04-06T03:27:26.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:26 smithi195 ceph-mon[53157]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-06T03:27:26.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:26 smithi195 ceph-mon[53157]: osdmap e115: 8 total, 7 up, 8 in 2024-04-06T03:27:27.114 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:27 smithi060 podman[80960]: 2024-04-06 03:27:27.023975813 +0000 UTC m=+0.541325091 container init 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:27:27.114 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:27 smithi060 podman[80960]: 2024-04-06 03:27:27.032246222 +0000 UTC m=+0.549595491 container start 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-06T03:27:27.365 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:27 smithi060 podman[80960]: 2024-04-06 03:27:27.145812618 +0000 UTC m=+0.663161892 container attach 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-06T03:27:27.568 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:27 smithi060 podman[80960]: 2024-04-06 03:27:27.426003526 +0000 UTC m=+0.943352845 container died 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:27:27.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 246897 10.42 MB/sec execute 451 sec latency 49.938 ms 2024-04-06T03:27:28.595 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:28.595 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:28.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 247153 10.42 MB/sec execute 452 sec latency 57.380 ms 2024-04-06T03:27:28.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:28.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:28.846 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:28 smithi060 podman[81017]: 2024-04-06 03:27:28.596022962 +0000 UTC m=+1.163792873 container remove 123f1e42c8e1e5f4398f6a77e33c4a23c7050260533a6ceaf507368664f7744a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-06T03:27:28.846 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:28 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-04-06T03:27:28.847 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:28 smithi060 systemd[1]: Stopped Ceph osd.3 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:27:28.847 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:28 smithi060 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.3.service: Consumed 5min 11.228s CPU time. 2024-04-06T03:27:29.099 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:28 smithi060 systemd[1]: Starting Ceph osd.3 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:27:29.369 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:29 smithi060 podman[81116]: 2024-04-06 03:27:29.055827279 +0000 UTC m=+0.020632897 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:27:29.369 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:29 smithi060 podman[81116]: 2024-04-06 03:27:29.230829517 +0000 UTC m=+0.195635122 container create e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:27:29.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:29 smithi060 ceph-mon[67116]: pgmap v179: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 10 peering, 11 stale+active+clean, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s wr, 427 op/s; 9/1617 objects degraded (0.557%) 2024-04-06T03:27:29.620 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:29 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 9/1617 objects degraded (0.557%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:29.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 247414 10.41 MB/sec execute 453 sec latency 83.291 ms 2024-04-06T03:27:29.721 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:29 smithi195 ceph-mon[53157]: pgmap v179: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 10 peering, 11 stale+active+clean, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 MiB/s wr, 427 op/s; 9/1617 objects degraded (0.557%) 2024-04-06T03:27:29.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:29 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 9/1617 objects degraded (0.557%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:29.871 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:29 smithi060 podman[81116]: 2024-04-06 03:27:29.626904514 +0000 UTC m=+0.591710120 container init e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-06T03:27:29.871 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:29 smithi060 podman[81116]: 2024-04-06 03:27:29.634216485 +0000 UTC m=+0.599022099 container start e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-06T03:27:29.871 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:29 smithi060 podman[81116]: 2024-04-06 03:27:29.815486208 +0000 UTC m=+0.780291817 container attach e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:27:30.375 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 bash[81116]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate[81130]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-06T03:27:30.376 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:30 smithi060 podman[81116]: 2024-04-06 03:27:30.216342765 +0000 UTC m=+1.181148371 container died e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-06T03:27:30.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 247678 10.40 MB/sec execute 454 sec latency 48.799 ms 2024-04-06T03:27:31.397 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:31 smithi060 ceph-mon[67116]: pgmap v180: 97 pgs: 11 active+undersized, 20 active+undersized+degraded, 10 peering, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 390 op/s; 189/1680 objects degraded (11.250%) 2024-04-06T03:27:31.397 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 podman[81251]: 2024-04-06 03:27:31.222288898 +0000 UTC m=+0.996178740 container remove e29767eacf47b7a42c5b22c2788d68a4b631853b5b81c769dd88be41c9bd8d8f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:27:31.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 247972 10.39 MB/sec execute 455 sec latency 87.688 ms 2024-04-06T03:27:31.662 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 podman[81307]: 2024-04-06 03:27:31.369422521 +0000 UTC m=+0.024800752 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:27:31.662 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 podman[81307]: 2024-04-06 03:27:31.521582235 +0000 UTC m=+0.176960456 container create 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-06T03:27:31.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:31 smithi195 ceph-mon[53157]: pgmap v180: 97 pgs: 11 active+undersized, 20 active+undersized+degraded, 10 peering, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 390 op/s; 189/1680 objects degraded (11.250%) 2024-04-06T03:27:31.914 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 podman[81307]: 2024-04-06 03:27:31.866097966 +0000 UTC m=+0.521476185 container init 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-06T03:27:31.914 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 podman[81307]: 2024-04-06 03:27:31.872541431 +0000 UTC m=+0.527919646 container start 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-06T03:27:31.914 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 ceph-osd[81326]: -- 172.21.15.60:0/4190311689 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c1bfcb680 con 0x555c1bf95000 2024-04-06T03:27:32.170 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:31 smithi060 bash[81307]: 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb 2024-04-06T03:27:32.170 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:32 smithi060 systemd[1]: Started Ceph osd.3 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:27:32.660 INFO:teuthology.orchestra.run.smithi060.stdout: 5 248258 10.38 MB/sec execute 456 sec latency 63.970 ms 2024-04-06T03:27:32.692 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:32 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:27:32.441+0000 7ff3f30527c0 -1 Falling back to public interface 2024-04-06T03:27:33.194 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:33 smithi060 ceph-mon[67116]: pgmap v181: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 MiB/s wr, 328 op/s; 216/1656 objects degraded (13.043%) 2024-04-06T03:27:33.194 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:33.194 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:33.194 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:33.194 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:33 smithi060 ceph-mon[67116]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-06T03:27:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:33 smithi195 ceph-mon[53157]: pgmap v181: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 MiB/s wr, 328 op/s; 216/1656 objects degraded (13.043%) 2024-04-06T03:27:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:33 smithi195 ceph-mon[53157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-06T03:27:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 248535 10.37 MB/sec execute 457 sec latency 93.290 ms 2024-04-06T03:27:34.218 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:34 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 214/1680 objects degraded (12.738%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:34.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:34 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 214/1680 objects degraded (12.738%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 248915 10.37 MB/sec execute 458 sec latency 42.706 ms 2024-04-06T03:27:35.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:35 smithi060 ceph-mon[67116]: pgmap v182: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 481 op/s; 214/1680 objects degraded (12.738%) 2024-04-06T03:27:35.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:35 smithi195 ceph-mon[53157]: pgmap v182: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 481 op/s; 214/1680 objects degraded (12.738%) 2024-04-06T03:27:35.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 249176 10.36 MB/sec execute 459 sec latency 63.717 ms 2024-04-06T03:27:36.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 249564 10.37 MB/sec execute 460 sec latency 66.080 ms 2024-04-06T03:27:37.308 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:37 smithi060 ceph-mon[67116]: pgmap v183: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 449 op/s; 212/1644 objects degraded (12.895%) 2024-04-06T03:27:37.471 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:37 smithi195 ceph-mon[53157]: pgmap v183: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 449 op/s; 212/1644 objects degraded (12.895%) 2024-04-06T03:27:37.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 249888 10.36 MB/sec execute 461 sec latency 41.421 ms 2024-04-06T03:27:38.591 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:38 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:27:38.347+0000 7ff3f30527c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-06T03:27:38.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 250228 10.37 MB/sec execute 462 sec latency 60.645 ms 2024-04-06T03:27:39.349 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:39 smithi060 ceph-mon[67116]: pgmap v184: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 376 op/s; 211/1641 objects degraded (12.858%) 2024-04-06T03:27:39.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:39 smithi195 ceph-mon[53157]: pgmap v184: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 376 op/s; 211/1641 objects degraded (12.858%) 2024-04-06T03:27:39.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 250458 10.36 MB/sec execute 463 sec latency 51.345 ms 2024-04-06T03:27:40.366 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:40 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 206/1641 objects degraded (12.553%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:40.366 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:40.366 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:40 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:27:40.245+0000 7ff3f30527c0 -1 osd.3 113 log_to_monitors true 2024-04-06T03:27:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:40 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 206/1641 objects degraded (12.553%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:40.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:40.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 250733 10.36 MB/sec execute 464 sec latency 42.554 ms 2024-04-06T03:27:41.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:41 smithi060 ceph-mon[67116]: pgmap v185: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 446 op/s; 206/1641 objects degraded (12.553%) 2024-04-06T03:27:41.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:41 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:41.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:41 smithi060 ceph-mon[67116]: from='osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:27:41.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:41 smithi060 ceph-mon[67116]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:27:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:41 smithi195 ceph-mon[53157]: pgmap v185: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 446 op/s; 206/1641 objects degraded (12.553%) 2024-04-06T03:27:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:41 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:41 smithi195 ceph-mon[53157]: from='osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:27:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:41 smithi195 ceph-mon[53157]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:27:41.652 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:27:41 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:27:41.564+0000 7ff3e04a2700 -1 osd.3 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:27:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 250986 10.34 MB/sec execute 465 sec latency 58.667 ms 2024-04-06T03:27:42.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:27:42.405 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:27:42.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: osdmap e116: 8 total, 7 up, 8 in 2024-04-06T03:27:42.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:27:42.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='osd.3 ' entity='osd.3' 2024-04-06T03:27:42.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:42.406 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: osdmap e116: 8 total, 7 up, 8 in 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='osd.3 ' entity='osd.3' 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:42.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251222 10.34 MB/sec execute 466 sec latency 126.184 ms 2024-04-06T03:27:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251222 10.32 MB/sec execute 467 sec latency 1126.216 ms 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: pgmap v187: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 531 op/s; 208/1668 objects degraded (12.470%) 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019] boot 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: osdmap e117: 8 total, 8 up, 8 in 2024-04-06T03:27:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: pgmap v187: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 531 op/s; 208/1668 objects degraded (12.470%) 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: osd.3 [v2:172.21.15.60:6826/1235279019,v1:172.21.15.60:6827/1235279019] boot 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: osdmap e117: 8 total, 8 up, 8 in 2024-04-06T03:27:43.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:27:44.647 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:44 smithi060 ceph-mon[67116]: osdmap e118: 8 total, 8 up, 8 in 2024-04-06T03:27:44.647 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:44 smithi060 ceph-mon[67116]: pgmap v190: 97 pgs: 10 peering, 10 active+undersized, 21 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 572 op/s; 162/1632 objects degraded (9.926%) 2024-04-06T03:27:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251223 10.29 MB/sec execute 468 sec latency 2121.542 ms 2024-04-06T03:27:44.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:44 smithi195 ceph-mon[53157]: osdmap e118: 8 total, 8 up, 8 in 2024-04-06T03:27:44.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:44 smithi195 ceph-mon[53157]: pgmap v190: 97 pgs: 10 peering, 10 active+undersized, 21 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 572 op/s; 162/1632 objects degraded (9.926%) 2024-04-06T03:27:45.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251397 10.28 MB/sec execute 469 sec latency 2153.543 ms 2024-04-06T03:27:45.670 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:45 smithi060 ceph-mon[67116]: osdmap e119: 8 total, 8 up, 8 in 2024-04-06T03:27:45.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:45 smithi195 ceph-mon[53157]: osdmap e119: 8 total, 8 up, 8 in 2024-04-06T03:27:46.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251632 10.27 MB/sec execute 470 sec latency 987.591 ms 2024-04-06T03:27:46.693 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:46 smithi060 ceph-mon[67116]: osdmap e120: 8 total, 8 up, 8 in 2024-04-06T03:27:46.693 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:46 smithi060 ceph-mon[67116]: pgmap v193: 97 pgs: 4 remapped+peering, 10 peering, 6 active+undersized, 12 active+undersized+degraded, 65 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.5 MiB/s wr, 109 op/s; 85/1647 objects degraded (5.161%); 19 KiB/s, 4 objects/s recovering 2024-04-06T03:27:46.693 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:46 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 162/1632 objects degraded (9.926%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:46.693 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:46 smithi060 ceph-mon[67116]: osdmap e121: 8 total, 8 up, 8 in 2024-04-06T03:27:46.873 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:46 smithi195 ceph-mon[53157]: osdmap e120: 8 total, 8 up, 8 in 2024-04-06T03:27:46.873 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:46 smithi195 ceph-mon[53157]: pgmap v193: 97 pgs: 4 remapped+peering, 10 peering, 6 active+undersized, 12 active+undersized+degraded, 65 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.5 MiB/s wr, 109 op/s; 85/1647 objects degraded (5.161%); 19 KiB/s, 4 objects/s recovering 2024-04-06T03:27:46.873 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:46 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 162/1632 objects degraded (9.926%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:27:46.873 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:46 smithi195 ceph-mon[53157]: osdmap e121: 8 total, 8 up, 8 in 2024-04-06T03:27:47.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 251986 10.26 MB/sec execute 471 sec latency 997.486 ms 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: pgmap v195: 97 pgs: 4 remapped+peering, 10 peering, 83 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 332 op/s; 2.5 MiB/s, 36 objects/s recovering 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: pgmap v196: 97 pgs: 4 remapped+peering, 10 peering, 83 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 334 op/s; 2.5 MiB/s, 36 objects/s recovering 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: osdmap e122: 8 total, 8 up, 8 in 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.664 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.665 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:27:47.665 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:27:47.665 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:47 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: pgmap v195: 97 pgs: 4 remapped+peering, 10 peering, 83 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 332 op/s; 2.5 MiB/s, 36 objects/s recovering 2024-04-06T03:27:47.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: pgmap v196: 97 pgs: 4 remapped+peering, 10 peering, 83 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 334 op/s; 2.5 MiB/s, 36 objects/s recovering 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: osdmap e122: 8 total, 8 up, 8 in 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:27:47.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:47 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-06T03:27:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 252365 10.25 MB/sec execute 472 sec latency 988.045 ms 2024-04-06T03:27:48.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:48 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1647 objects degraded (5.161%), 12 pgs degraded) 2024-04-06T03:27:48.747 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:48 smithi060 ceph-mon[67116]: osdmap e123: 8 total, 8 up, 8 in 2024-04-06T03:27:48.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:48 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1647 objects degraded (5.161%), 12 pgs degraded) 2024-04-06T03:27:48.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:48 smithi195 ceph-mon[53157]: osdmap e123: 8 total, 8 up, 8 in 2024-04-06T03:27:49.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 252881 10.26 MB/sec execute 473 sec latency 34.801 ms 2024-04-06T03:27:49.752 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:49 smithi060 ceph-mon[67116]: pgmap v199: 97 pgs: 4 remapped+peering, 93 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.7 MiB/s wr, 521 op/s; 3.8 MiB/s, 47 objects/s recovering 2024-04-06T03:27:49.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:49 smithi195 ceph-mon[53157]: pgmap v199: 97 pgs: 4 remapped+peering, 93 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.7 MiB/s wr, 521 op/s; 3.8 MiB/s, 47 objects/s recovering 2024-04-06T03:27:50.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 253288 10.27 MB/sec execute 474 sec latency 47.335 ms 2024-04-06T03:27:50.908 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:27:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 253643 10.27 MB/sec execute 475 sec latency 38.947 ms 2024-04-06T03:27:51.800 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:51 smithi060 ceph-mon[67116]: pgmap v200: 97 pgs: 1 remapped+peering, 96 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.1 MiB/s wr, 362 op/s; 2.8 MiB/s, 41 objects/s recovering 2024-04-06T03:27:51.800 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:51 smithi060 ceph-mon[67116]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:51.966 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:27:51.966 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (7m) 11s ago 16m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (3m) 11s ago 16m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (3m) 3m ago 15m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 11s ago 16m - - 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (10m) 11s ago 10m 41.1M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (10m) 3m ago 10m 15.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (6m) 11s ago 17m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (5m) 3m ago 15m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (4m) 11s ago 17m 51.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (4m) 3m ago 15m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (10m) 11s ago 10m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (10m) 3m ago 10m 81.2M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (6m) 11s ago 16m 9470k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (6m) 3m ago 15m 21.3M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (2m) 11s ago 14m 332M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:27:51.967 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (115s) 11s ago 14m 292M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (69s) 11s ago 13m 214M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (20s) 11s ago 13m 14.8M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (13m) 3m ago 13m 562M 2704M 17.2.0 e1d6a67b021e e5b49c7c932c 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (12m) 3m ago 12m 433M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (12m) 3m ago 12m 342M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (11m) 3m ago 11m 570M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:27:51.968 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (5m) 11s ago 15m 63.2M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:27:51.971 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:51 smithi195 ceph-mon[53157]: pgmap v200: 97 pgs: 1 remapped+peering, 96 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.1 MiB/s wr, 362 op/s; 2.8 MiB/s, 41 objects/s recovering 2024-04-06T03:27:51.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:51 smithi195 ceph-mon[53157]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-06T03:27:52.618 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:27:52.619 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:27:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: 5 253901 10.27 MB/sec execute 476 sec latency 43.436 ms 2024-04-06T03:27:52.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:52 smithi060 ceph-mon[67116]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:52.818 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:52 smithi060 ceph-mon[67116]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:52.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:52 smithi195 ceph-mon[53157]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:52.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:52 smithi195 ceph-mon[53157]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:53.150 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:27:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:27:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "10/23 daemons upgraded", 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:27:53.151 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:27:53.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 254185 10.27 MB/sec execute 477 sec latency 45.259 ms 2024-04-06T03:27:53.773 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:27:53.773 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:27:53.773 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:27:53.833 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:53 smithi060 ceph-mon[67116]: pgmap v201: 97 pgs: 97 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.5 MiB/s wr, 224 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-04-06T03:27:53.833 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:53 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/861274705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:53.833 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:53 smithi060 ceph-mon[67116]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:53 smithi195 ceph-mon[53157]: pgmap v201: 97 pgs: 97 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.5 MiB/s wr, 224 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-04-06T03:27:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:53 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/861274705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:27:53.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:53 smithi195 ceph-mon[53157]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:27:54.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 254479 10.27 MB/sec execute 478 sec latency 45.256 ms 2024-04-06T03:27:54.855 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:54 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3716542042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:27:54.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:54 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3716542042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:27:55.609 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:55 smithi060 ceph-mon[67116]: pgmap v202: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.3 MiB/s wr, 357 op/s; 885 KiB/s, 16 objects/s recovering 2024-04-06T03:27:55.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 254719 10.25 MB/sec execute 479 sec latency 49.971 ms 2024-04-06T03:27:55.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:55 smithi195 ceph-mon[53157]: pgmap v202: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.3 MiB/s wr, 357 op/s; 885 KiB/s, 16 objects/s recovering 2024-04-06T03:27:56.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 254964 10.24 MB/sec execute 480 sec latency 61.494 ms 2024-04-06T03:27:57.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 255289 10.23 MB/sec execute 481 sec latency 50.470 ms 2024-04-06T03:27:58.409 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:58 smithi060 ceph-mon[67116]: pgmap v203: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.8 MiB/s wr, 486 op/s; 712 KiB/s, 13 objects/s recovering 2024-04-06T03:27:58.409 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:58.409 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:27:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 255591 10.22 MB/sec execute 482 sec latency 58.212 ms 2024-04-06T03:27:58.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:58 smithi195 ceph-mon[53157]: pgmap v203: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.8 MiB/s wr, 486 op/s; 712 KiB/s, 13 objects/s recovering 2024-04-06T03:27:58.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:27:58.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:27:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 255901 10.22 MB/sec execute 483 sec latency 72.319 ms 2024-04-06T03:28:00.492 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:00 smithi060 ceph-mon[67116]: pgmap v204: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.0 MiB/s wr, 502 op/s; 647 KiB/s, 12 objects/s recovering 2024-04-06T03:28:00.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 256214 10.21 MB/sec execute 484 sec latency 50.681 ms 2024-04-06T03:28:00.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:00 smithi195 ceph-mon[53157]: pgmap v204: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.0 MiB/s wr, 502 op/s; 647 KiB/s, 12 objects/s recovering 2024-04-06T03:28:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 256593 10.22 MB/sec execute 485 sec latency 59.877 ms 2024-04-06T03:28:02.563 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:02 smithi060 ceph-mon[67116]: pgmap v205: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 421 op/s; 249 KiB/s, 6 objects/s recovering 2024-04-06T03:28:02.563 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:28:02.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 257133 10.24 MB/sec execute 486 sec latency 34.152 ms 2024-04-06T03:28:02.716 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:02 smithi195 ceph-mon[53157]: pgmap v205: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 421 op/s; 249 KiB/s, 6 objects/s recovering 2024-04-06T03:28:02.717 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:28:03.574 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:03 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:28:03.574 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:03 smithi060 ceph-mon[67116]: Upgrade: osd.4 is safe to restart 2024-04-06T03:28:03.574 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:03.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 257548 10.24 MB/sec execute 487 sec latency 34.773 ms 2024-04-06T03:28:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:03 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T03:28:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:03 smithi195 ceph-mon[53157]: Upgrade: osd.4 is safe to restart 2024-04-06T03:28:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:04.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:04 smithi060 ceph-mon[67116]: pgmap v206: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 34 KiB/s, 1 objects/s recovering 2024-04-06T03:28:04.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:04 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:04.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:04 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:28:04.581 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:04 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:04.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:04 smithi195 ceph-mon[53157]: pgmap v206: 97 pgs: 97 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 34 KiB/s, 1 objects/s recovering 2024-04-06T03:28:04.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:04 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:04.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:04 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:28:04.619 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:04 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:04.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 257995 10.24 MB/sec execute 488 sec latency 30.982 ms 2024-04-06T03:28:05.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:05 smithi060 ceph-mon[67116]: Upgrade: Updating osd.4 2024-04-06T03:28:05.569 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:05 smithi060 ceph-mon[67116]: Deploying daemon osd.4 on smithi195 2024-04-06T03:28:05.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 258462 10.25 MB/sec execute 489 sec latency 48.811 ms 2024-04-06T03:28:05.686 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:05 smithi195 ceph-mon[53157]: Upgrade: Updating osd.4 2024-04-06T03:28:05.686 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:05 smithi195 ceph-mon[53157]: Deploying daemon osd.4 on smithi195 2024-04-06T03:28:06.476 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:06 smithi195 ceph-mon[53157]: pgmap v207: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.9 MiB/s wr, 620 op/s 2024-04-06T03:28:06.624 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:06 smithi060 ceph-mon[67116]: pgmap v207: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.9 MiB/s wr, 620 op/s 2024-04-06T03:28:06.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 258770 10.24 MB/sec execute 490 sec latency 65.933 ms 2024-04-06T03:28:07.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 259053 10.23 MB/sec execute 491 sec latency 84.095 ms 2024-04-06T03:28:07.972 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:07 smithi195 systemd[1]: Stopping Ceph osd.4 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:08.404 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[31403]: 2024-04-06T03:28:08.140+0000 7f05c7456700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:08.405 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[31403]: 2024-04-06T03:28:08.140+0000 7f05c7456700 -1 osd.4 123 *** Got signal Terminated *** 2024-04-06T03:28:08.405 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[31403]: 2024-04-06T03:28:08.140+0000 7f05c7456700 -1 osd.4 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:28:08.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:08 smithi060 ceph-mon[67116]: pgmap v208: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 518 op/s 2024-04-06T03:28:08.438 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:08 smithi060 ceph-mon[67116]: osd.4 marked itself down and dead 2024-04-06T03:28:08.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 259315 10.21 MB/sec execute 492 sec latency 243.474 ms 2024-04-06T03:28:08.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:08 smithi195 ceph-mon[53157]: pgmap v208: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 518 op/s 2024-04-06T03:28:08.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:08 smithi195 ceph-mon[53157]: osd.4 marked itself down and dead 2024-04-06T03:28:09.357 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:09 smithi195 podman[57649]: 2024-04-06 03:28:09.047431685 +0000 UTC m=+1.046282902 container died e5b49c7c932ceeed5af61861b31c45f02ebef576347645a7fe96f975fd24dfd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8) 2024-04-06T03:28:09.662 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:09 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:28:09.662 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:09 smithi060 ceph-mon[67116]: osdmap e124: 8 total, 7 up, 8 in 2024-04-06T03:28:09.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 259384 10.20 MB/sec execute 493 sec latency 1056.380 ms 2024-04-06T03:28:09.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:09 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:28:09.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:09 smithi195 ceph-mon[53157]: osdmap e124: 8 total, 7 up, 8 in 2024-04-06T03:28:09.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:09 smithi195 podman[57649]: 2024-04-06 03:28:09.448800249 +0000 UTC m=+1.447651458 container cleanup e5b49c7c932ceeed5af61861b31c45f02ebef576347645a7fe96f975fd24dfd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, io.k8s.display-name=CentOS Stream 8, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_CLEAN=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-06T03:28:09.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:09 smithi195 bash[57649]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T03:28:10.151 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:09 smithi195 podman[57666]: 2024-04-06 03:28:09.758880471 +0000 UTC m=+0.713820031 container remove e5b49c7c932ceeed5af61861b31c45f02ebef576347645a7fe96f975fd24dfd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, architecture=x86_64, RELEASE=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-06T03:28:10.412 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.055476271 +0000 UTC m=+0.021131765 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:10.412 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.168240503 +0000 UTC m=+0.133895991 container create 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, RELEASE=squid-9f19235, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:28:10.432 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:10 smithi060 ceph-mon[67116]: pgmap v210: 97 pgs: 8 peering, 13 stale+active+clean, 76 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-06T03:28:10.432 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:10 smithi060 ceph-mon[67116]: osdmap e125: 8 total, 7 up, 8 in 2024-04-06T03:28:10.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 259723 10.20 MB/sec execute 494 sec latency 59.539 ms 2024-04-06T03:28:10.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:10 smithi195 ceph-mon[53157]: pgmap v210: 97 pgs: 8 peering, 13 stale+active+clean, 76 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-06T03:28:10.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:10 smithi195 ceph-mon[53157]: osdmap e125: 8 total, 7 up, 8 in 2024-04-06T03:28:10.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.436463095 +0000 UTC m=+0.402118596 container init 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-06T03:28:10.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.441415889 +0000 UTC m=+0.407071386 container start 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:10.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.54870586 +0000 UTC m=+0.514361358 container attach 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-06T03:28:11.222 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:10 smithi195 podman[57752]: 2024-04-06 03:28:10.807775178 +0000 UTC m=+0.773430657 container died 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-06T03:28:11.636 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:11 smithi195 ceph-mon[53157]: pgmap v212: 97 pgs: 3 active+undersized, 11 active+undersized+degraded, 8 peering, 10 stale+active+clean, 65 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 671 op/s; 93/1638 objects degraded (5.678%) 2024-04-06T03:28:11.637 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:11 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 93/1638 objects degraded (5.678%), 11 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 260070 10.20 MB/sec execute 495 sec latency 66.261 ms 2024-04-06T03:28:11.723 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:11 smithi060 ceph-mon[67116]: pgmap v212: 97 pgs: 3 active+undersized, 11 active+undersized+degraded, 8 peering, 10 stale+active+clean, 65 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 671 op/s; 93/1638 objects degraded (5.678%) 2024-04-06T03:28:11.723 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:11 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 93/1638 objects degraded (5.678%), 11 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:11.900 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:11 smithi195 podman[57778]: 2024-04-06 03:28:11.637563751 +0000 UTC m=+0.819786561 container remove 9c8fbaa3dab813ddf2204d0eb9af617ba8cf16cfc91e2a2729fba199726a09fa (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:28:11.900 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:11 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.4.service: Deactivated successfully. 2024-04-06T03:28:11.901 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:11 smithi195 systemd[1]: Stopped Ceph osd.4 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:11.901 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:11 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.4.service: Consumed 4min 59.207s CPU time. 2024-04-06T03:28:12.162 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:11 smithi195 systemd[1]: Starting Ceph osd.4 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:12.163 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 podman[57876]: 2024-04-06 03:28:12.036905195 +0000 UTC m=+0.020218209 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:12.424 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 podman[57876]: 2024-04-06 03:28:12.163552719 +0000 UTC m=+0.146865721 container create 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T03:28:12.424 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 podman[57876]: 2024-04-06 03:28:12.424893417 +0000 UTC m=+0.408206421 container init 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235) 2024-04-06T03:28:12.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 260401 10.20 MB/sec execute 496 sec latency 52.975 ms 2024-04-06T03:28:12.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 podman[57876]: 2024-04-06 03:28:12.42941373 +0000 UTC m=+0.412726739 container start 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:12.722 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 podman[57876]: 2024-04-06 03:28:12.516735011 +0000 UTC m=+0.500048020 container attach 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-06T03:28:13.271 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 bash[57876]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate[57890]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-06T03:28:13.272 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:13 smithi195 podman[57876]: 2024-04-06 03:28:13.008613847 +0000 UTC m=+0.991926857 container died 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-06T03:28:13.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:13.508 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:13.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 260707 10.20 MB/sec execute 497 sec latency 43.874 ms 2024-04-06T03:28:13.704 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:13.705 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:13.955 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:13 smithi195 podman[58006]: 2024-04-06 03:28:13.705383176 +0000 UTC m=+0.690947951 container remove 67fdcaf0384fa54b59dcee32a32e0fc81a0ed48d57c6703d44cf4ff83c16178c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-activate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0) 2024-04-06T03:28:13.956 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:13 smithi195 podman[58065]: 2024-04-06 03:28:13.834212278 +0000 UTC m=+0.020277066 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:14.220 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:13 smithi195 podman[58065]: 2024-04-06 03:28:13.956517597 +0000 UTC m=+0.142582368 container create eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS) 2024-04-06T03:28:14.221 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 podman[58065]: 2024-04-06 03:28:14.192525414 +0000 UTC m=+0.378590200 container init eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-06T03:28:14.221 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 podman[58065]: 2024-04-06 03:28:14.196476582 +0000 UTC m=+0.382541353 container start eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-06T03:28:14.221 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 ceph-osd[58084]: -- 172.21.15.195:0/1782488155 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635c0553680 con 0x5635c051cc00 2024-04-06T03:28:14.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:14 smithi195 ceph-mon[53157]: pgmap v213: 97 pgs: 8 active+undersized, 13 active+undersized+degraded, 8 peering, 5 stale+active+clean, 63 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 357 op/s; 120/1611 objects degraded (7.449%) 2024-04-06T03:28:14.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:14 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:14.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:14 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:14.472 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 bash[58065]: eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 2024-04-06T03:28:14.472 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 systemd[1]: Started Ceph osd.4 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:14.518 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:14 smithi060 ceph-mon[67116]: pgmap v213: 97 pgs: 8 active+undersized, 13 active+undersized+degraded, 8 peering, 5 stale+active+clean, 63 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 357 op/s; 120/1611 objects degraded (7.449%) 2024-04-06T03:28:14.518 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:14 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:14.518 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:14 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:14.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 260904 10.20 MB/sec execute 498 sec latency 60.621 ms 2024-04-06T03:28:15.222 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:28:14.739+0000 7f278aa397c0 -1 Falling back to public interface 2024-04-06T03:28:15.531 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 261221 10.20 MB/sec execute 499 sec latency 65.580 ms 2024-04-06T03:28:15.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:16.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:16 smithi195 ceph-mon[53157]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.7 MiB/s wr, 521 op/s; 261/1596 objects degraded (16.353%) 2024-04-06T03:28:16.537 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:16 smithi060 ceph-mon[67116]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 327 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.7 MiB/s wr, 521 op/s; 261/1596 objects degraded (16.353%) 2024-04-06T03:28:16.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 261536 10.19 MB/sec execute 500 sec latency 51.119 ms 2024-04-06T03:28:17.564 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:17 smithi060 ceph-mon[67116]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.2 MiB/s wr, 311 op/s; 266/1650 objects degraded (16.121%) 2024-04-06T03:28:17.564 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:17 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 266/1650 objects degraded (16.121%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:17.662 INFO:teuthology.orchestra.run.smithi060.stdout: 5 261796 10.18 MB/sec execute 501 sec latency 78.556 ms 2024-04-06T03:28:17.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:17 smithi195 ceph-mon[53157]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.2 MiB/s wr, 311 op/s; 266/1650 objects degraded (16.121%) 2024-04-06T03:28:17.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:17 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 266/1650 objects degraded (16.121%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:18.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 262025 10.17 MB/sec execute 502 sec latency 57.011 ms 2024-04-06T03:28:19.584 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:19.584 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:19.584 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:19 smithi060 ceph-mon[67116]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 406 op/s; 263/1674 objects degraded (15.711%) 2024-04-06T03:28:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 262283 10.16 MB/sec execute 503 sec latency 56.842 ms 2024-04-06T03:28:19.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:19.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:19.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:19 smithi195 ceph-mon[53157]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 406 op/s; 263/1674 objects degraded (15.711%) 2024-04-06T03:28:20.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 262517 10.15 MB/sec execute 504 sec latency 83.115 ms 2024-04-06T03:28:20.867 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:28:20.607+0000 7f278aa397c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-06T03:28:20.933 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:20.933 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:21.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:21.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:21.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 262741 10.14 MB/sec execute 505 sec latency 83.305 ms 2024-04-06T03:28:21.949 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:21 smithi060 ceph-mon[67116]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.3 MiB/s wr, 387 op/s; 263/1689 objects degraded (15.571%) 2024-04-06T03:28:22.068 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:21 smithi195 ceph-mon[53157]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.3 MiB/s wr, 387 op/s; 263/1689 objects degraded (15.571%) 2024-04-06T03:28:22.335 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:28:22.216+0000 7f278aa397c0 -1 osd.4 123 log_to_monitors true 2024-04-06T03:28:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 263069 10.14 MB/sec execute 506 sec latency 46.455 ms 2024-04-06T03:28:22.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:22 smithi060 ceph-mon[67116]: from='osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:28:22.986 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:22 smithi060 ceph-mon[67116]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:28:23.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:22 smithi195 ceph-mon[53157]: from='osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:28:23.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:22 smithi195 ceph-mon[53157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:28:23.223 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:28:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:28:22.945+0000 7f277be91700 -1 osd.4 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:28:23.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 263475 10.14 MB/sec execute 507 sec latency 62.561 ms 2024-04-06T03:28:24.025 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 347 op/s; 262/1650 objects degraded (15.879%) 2024-04-06T03:28:24.025 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 262/1650 objects degraded (15.879%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: osdmap e126: 8 total, 7 up, 8 in 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.026 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:23 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 347 op/s; 262/1650 objects degraded (15.879%) 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 262/1650 objects degraded (15.879%), 28 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: osdmap e126: 8 total, 7 up, 8 in 2024-04-06T03:28:24.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:23 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:24.238 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:28:24.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 263640 10.13 MB/sec execute 508 sec latency 695.478 ms 2024-04-06T03:28:24.910 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:28:24.910 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (7m) 44s ago 16m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:28:24.910 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (3m) 44s ago 16m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:28:24.910 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (3m) 6s ago 16m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:28:24.910 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 44s ago 16m - - 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (10m) 44s ago 10m 41.1M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (10m) 6s ago 10m 16.2M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (6m) 44s ago 18m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (5m) 6s ago 16m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (5m) 44s ago 18m 51.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (4m) 6s ago 16m 44.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (10m) 44s ago 10m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (10m) 6s ago 10m 83.6M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (7m) 44s ago 16m 9470k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (6m) 6s ago 15m 18.7M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (3m) 44s ago 15m 332M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (2m) 44s ago 14m 292M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:28:24.911 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (102s) 44s ago 14m 214M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (53s) 44s ago 14m 14.8M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (11s) 6s ago 13m 12.6M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (13m) 6s ago 13m 625M 2704M 17.2.0 e1d6a67b021e aac7709c4bf4 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (12m) 6s ago 12m 543M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (12m) 6s ago 12m 592M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:28:24.912 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (5m) 44s ago 15m 63.2M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:28:25.161 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: pgmap v220: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.9 MiB/s wr, 417 op/s; 259/1578 objects degraded (16.413%) 2024-04-06T03:28:25.161 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037] boot 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: osdmap e127: 8 total, 8 up, 8 in 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:25.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:24 smithi060 ceph-mon[67116]: osdmap e128: 8 total, 8 up, 8 in 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: pgmap v220: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.9 MiB/s wr, 417 op/s; 259/1578 objects degraded (16.413%) 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: osd.4 [v2:172.21.15.195:6800/335855037,v1:172.21.15.195:6801/335855037] boot 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: osdmap e127: 8 total, 8 up, 8 in 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:28:25.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:24 smithi195 ceph-mon[53157]: osdmap e128: 8 total, 8 up, 8 in 2024-04-06T03:28:25.525 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:28:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:28:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T03:28:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 9 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:28:25.527 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:28:25.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 263817 10.11 MB/sec execute 509 sec latency 1695.507 ms 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:28:25.870 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "11/23 daemons upgraded", 2024-04-06T03:28:25.871 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:28:25.871 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:28:25.871 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:28:26.124 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:25 smithi060 ceph-mon[67116]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:26.124 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:25 smithi060 ceph-mon[67116]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:26.124 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:25 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1256263146' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:26.124 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:25 smithi060 ceph-mon[67116]: osdmap e129: 8 total, 8 up, 8 in 2024-04-06T03:28:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:25 smithi195 ceph-mon[53157]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:25 smithi195 ceph-mon[53157]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:25 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1256263146' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:26.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:25 smithi195 ceph-mon[53157]: osdmap e129: 8 total, 8 up, 8 in 2024-04-06T03:28:26.431 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 189/1563 objects degraded (12.092%), 20 pgs degraded 2024-04-06T03:28:26.431 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:28:26.431 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:28:26.431 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 189/1563 objects degraded (12.092%), 20 pgs degraded 2024-04-06T03:28:26.431 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.f is active+undersized+degraded, acting [0,2] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-04-06T03:28:26.432 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-04-06T03:28:26.433 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-04-06T03:28:26.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 264005 10.11 MB/sec execute 510 sec latency 2123.944 ms 2024-04-06T03:28:27.152 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:26 smithi060 ceph-mon[67116]: pgmap v223: 97 pgs: 11 peering, 10 active+undersized, 20 active+undersized+degraded, 56 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.7 MiB/s wr, 342 op/s; 189/1563 objects degraded (12.092%) 2024-04-06T03:28:27.152 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:26 smithi060 ceph-mon[67116]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:27.152 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:26 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/4148445234' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:28:27.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:26 smithi195 ceph-mon[53157]: pgmap v223: 97 pgs: 11 peering, 10 active+undersized, 20 active+undersized+degraded, 56 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.7 MiB/s wr, 342 op/s; 189/1563 objects degraded (12.092%) 2024-04-06T03:28:27.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:26 smithi195 ceph-mon[53157]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:27.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:26 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/4148445234' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:28:27.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 264433 10.12 MB/sec execute 511 sec latency 42.360 ms 2024-04-06T03:28:28.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:28.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:28.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 264789 10.12 MB/sec execute 512 sec latency 37.787 ms 2024-04-06T03:28:28.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:28.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:29.429 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:29 smithi060 ceph-mon[67116]: pgmap v225: 97 pgs: 1 active+recovery_wait+degraded, 11 peering, 9 active+undersized, 19 active+undersized+degraded, 57 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.7 MiB/s wr, 502 op/s; 192/1569 objects degraded (12.237%); 2.5 KiB/s, 0 objects/s recovering 2024-04-06T03:28:29.429 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:29 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 192/1569 objects degraded (12.237%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:29.429 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:29 smithi060 ceph-mon[67116]: osdmap e130: 8 total, 8 up, 8 in 2024-04-06T03:28:29.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 265075 10.12 MB/sec execute 513 sec latency 124.114 ms 2024-04-06T03:28:29.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:29 smithi195 ceph-mon[53157]: pgmap v225: 97 pgs: 1 active+recovery_wait+degraded, 11 peering, 9 active+undersized, 19 active+undersized+degraded, 57 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.7 MiB/s wr, 502 op/s; 192/1569 objects degraded (12.237%); 2.5 KiB/s, 0 objects/s recovering 2024-04-06T03:28:29.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:29 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 192/1569 objects degraded (12.237%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:29.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:29 smithi195 ceph-mon[53157]: osdmap e130: 8 total, 8 up, 8 in 2024-04-06T03:28:30.445 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:30 smithi060 ceph-mon[67116]: osdmap e131: 8 total, 8 up, 8 in 2024-04-06T03:28:30.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 265333 10.11 MB/sec execute 514 sec latency 881.684 ms 2024-04-06T03:28:30.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:30 smithi195 ceph-mon[53157]: osdmap e131: 8 total, 8 up, 8 in 2024-04-06T03:28:31.457 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:31 smithi060 ceph-mon[67116]: pgmap v228: 97 pgs: 1 active+recovery_wait+degraded, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 92 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 404 op/s; 60/1641 objects degraded (3.656%); 174 KiB/s, 41 keys/s, 51 objects/s recovering 2024-04-06T03:28:31.457 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:31 smithi060 ceph-mon[67116]: osdmap e132: 8 total, 8 up, 8 in 2024-04-06T03:28:31.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 265636 10.10 MB/sec execute 515 sec latency 317.745 ms 2024-04-06T03:28:31.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:31 smithi195 ceph-mon[53157]: pgmap v228: 97 pgs: 1 active+recovery_wait+degraded, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 92 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 404 op/s; 60/1641 objects degraded (3.656%); 174 KiB/s, 41 keys/s, 51 objects/s recovering 2024-04-06T03:28:31.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:31 smithi195 ceph-mon[53157]: osdmap e132: 8 total, 8 up, 8 in 2024-04-06T03:28:32.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:32 smithi060 ceph-mon[67116]: osdmap e133: 8 total, 8 up, 8 in 2024-04-06T03:28:32.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 265994 10.10 MB/sec execute 516 sec latency 52.108 ms 2024-04-06T03:28:32.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:32 smithi195 ceph-mon[53157]: osdmap e133: 8 total, 8 up, 8 in 2024-04-06T03:28:33.558 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:33 smithi060 ceph-mon[67116]: pgmap v231: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 5.7 MiB/s wr, 460 op/s; 40/1689 objects degraded (2.368%); 402 KiB/s, 51 keys/s, 67 objects/s recovering 2024-04-06T03:28:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 266327 10.09 MB/sec execute 517 sec latency 50.732 ms 2024-04-06T03:28:33.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:33 smithi195 ceph-mon[53157]: pgmap v231: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 93 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 5.7 MiB/s wr, 460 op/s; 40/1689 objects degraded (2.368%); 402 KiB/s, 51 keys/s, 67 objects/s recovering 2024-04-06T03:28:34.571 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:34 smithi060 ceph-mon[67116]: pgmap v232: 97 pgs: 1 peering, 96 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 576 op/s; 447 KiB/s, 37 keys/s, 41 objects/s recovering 2024-04-06T03:28:34.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 266703 10.09 MB/sec execute 518 sec latency 53.686 ms 2024-04-06T03:28:34.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:34 smithi195 ceph-mon[53157]: pgmap v232: 97 pgs: 1 peering, 96 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 576 op/s; 447 KiB/s, 37 keys/s, 41 objects/s recovering 2024-04-06T03:28:35.605 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:35 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1689 objects degraded (2.368%), 3 pgs degraded) 2024-04-06T03:28:35.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 267047 10.08 MB/sec execute 519 sec latency 54.180 ms 2024-04-06T03:28:35.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:35 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/1689 objects degraded (2.368%), 3 pgs degraded) 2024-04-06T03:28:36.626 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:36 smithi060 ceph-mon[67116]: pgmap v233: 97 pgs: 97 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 MiB/s wr, 586 op/s; 377 KiB/s, 31 keys/s, 22 objects/s recovering 2024-04-06T03:28:36.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 267449 10.10 MB/sec execute 520 sec latency 42.780 ms 2024-04-06T03:28:36.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:36 smithi195 ceph-mon[53157]: pgmap v233: 97 pgs: 97 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 MiB/s wr, 586 op/s; 377 KiB/s, 31 keys/s, 22 objects/s recovering 2024-04-06T03:28:37.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 267778 10.10 MB/sec execute 521 sec latency 49.045 ms 2024-04-06T03:28:38.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 268140 10.11 MB/sec execute 522 sec latency 44.603 ms 2024-04-06T03:28:38.921 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:38 smithi060 ceph-mon[67116]: pgmap v234: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 385 op/s; 227 KiB/s, 8 objects/s recovering 2024-04-06T03:28:39.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:38 smithi195 ceph-mon[53157]: pgmap v234: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 385 op/s; 227 KiB/s, 8 objects/s recovering 2024-04-06T03:28:39.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 268438 10.10 MB/sec execute 523 sec latency 63.643 ms 2024-04-06T03:28:39.936 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:39.937 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:39 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:39.937 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:39 smithi195 ceph-mon[53157]: Upgrade: osd.5 is safe to restart 2024-04-06T03:28:39.947 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:39.947 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:39 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T03:28:39.947 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:39 smithi060 ceph-mon[67116]: Upgrade: osd.5 is safe to restart 2024-04-06T03:28:40.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 268740 10.10 MB/sec execute 524 sec latency 46.752 ms 2024-04-06T03:28:40.961 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:40 smithi060 ceph-mon[67116]: pgmap v235: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 551 op/s; 192 KiB/s, 7 objects/s recovering 2024-04-06T03:28:40.962 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:40.962 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:28:40.962 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:40.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:40 smithi195 ceph-mon[53157]: pgmap v235: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 551 op/s; 192 KiB/s, 7 objects/s recovering 2024-04-06T03:28:40.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:40.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:28:40.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 269020 10.08 MB/sec execute 525 sec latency 72.593 ms 2024-04-06T03:28:41.967 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:41 smithi060 ceph-mon[67116]: Upgrade: Updating osd.5 2024-04-06T03:28:41.967 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:41 smithi060 ceph-mon[67116]: Deploying daemon osd.5 on smithi195 2024-04-06T03:28:42.176 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:41 smithi195 ceph-mon[53157]: Upgrade: Updating osd.5 2024-04-06T03:28:42.176 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:41 smithi195 ceph-mon[53157]: Deploying daemon osd.5 on smithi195 2024-04-06T03:28:42.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 269322 10.08 MB/sec execute 526 sec latency 72.861 ms 2024-04-06T03:28:43.392 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:43 smithi060 ceph-mon[67116]: pgmap v236: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 474 op/s; 100 KiB/s, 4 objects/s recovering 2024-04-06T03:28:43.392 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:43.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:43.393 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:43.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:43 smithi195 ceph-mon[53157]: pgmap v236: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 474 op/s; 100 KiB/s, 4 objects/s recovering 2024-04-06T03:28:43.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:43.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:43.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 269667 10.07 MB/sec execute 527 sec latency 42.701 ms 2024-04-06T03:28:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 270002 10.07 MB/sec execute 528 sec latency 60.749 ms 2024-04-06T03:28:45.061 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:44 smithi195 systemd[1]: Stopping Ceph osd.5 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:45.379 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:45 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[34374]: 2024-04-06T03:28:45.059+0000 7f67ef6b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:45.379 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:45 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[34374]: 2024-04-06T03:28:45.059+0000 7f67ef6b6700 -1 osd.5 133 *** Got signal Terminated *** 2024-04-06T03:28:45.380 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:45 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[34374]: 2024-04-06T03:28:45.059+0000 7f67ef6b6700 -1 osd.5 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:28:45.424 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:45 smithi060 ceph-mon[67116]: pgmap v237: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 504 op/s; 87 KiB/s, 4 objects/s recovering 2024-04-06T03:28:45.424 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:45 smithi060 ceph-mon[67116]: osd.5 marked itself down and dead 2024-04-06T03:28:45.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 270394 10.07 MB/sec execute 529 sec latency 290.252 ms 2024-04-06T03:28:45.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:45 smithi195 ceph-mon[53157]: pgmap v237: 97 pgs: 97 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.2 MiB/s wr, 504 op/s; 87 KiB/s, 4 objects/s recovering 2024-04-06T03:28:45.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:45 smithi195 ceph-mon[53157]: osd.5 marked itself down and dead 2024-04-06T03:28:46.335 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:46 smithi195 podman[59839]: 2024-04-06 03:28:46.028312846 +0000 UTC m=+1.102465983 container died aac7709c4bf4588e4d591125d25fd5b3266786d987a7adf94a5eae59f801f928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, distribution-scope=public, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, architecture=x86_64, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True) 2024-04-06T03:28:46.639 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:46 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:28:46.640 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:46 smithi060 ceph-mon[67116]: osdmap e134: 8 total, 7 up, 8 in 2024-04-06T03:28:46.640 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:46 smithi060 ceph-mon[67116]: osdmap e135: 8 total, 7 up, 8 in 2024-04-06T03:28:46.663 INFO:teuthology.orchestra.run.smithi060.stdout: 5 270535 10.06 MB/sec execute 530 sec latency 1045.156 ms 2024-04-06T03:28:46.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:46 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:28:46.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:46 smithi195 ceph-mon[53157]: osdmap e134: 8 total, 7 up, 8 in 2024-04-06T03:28:46.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:46 smithi195 ceph-mon[53157]: osdmap e135: 8 total, 7 up, 8 in 2024-04-06T03:28:46.722 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:46 smithi195 podman[59839]: 2024-04-06 03:28:46.435764291 +0000 UTC m=+1.509917431 container cleanup aac7709c4bf4588e4d591125d25fd5b3266786d987a7adf94a5eae59f801f928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, name=centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, RELEASE=HEAD, vcs-type=git, release=754, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , version=8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True) 2024-04-06T03:28:46.722 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:46 smithi195 bash[59839]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T03:28:47.390 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59855]: 2024-04-06 03:28:47.121043496 +0000 UTC m=+1.092246815 container remove aac7709c4bf4588e4d591125d25fd5b3266786d987a7adf94a5eae59f801f928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., vcs-type=git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754) 2024-04-06T03:28:47.396 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:47 smithi060 ceph-mon[67116]: pgmap v239: 97 pgs: 12 stale+active+clean, 85 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 407 op/s 2024-04-06T03:28:47.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 270974 10.07 MB/sec execute 531 sec latency 41.671 ms 2024-04-06T03:28:47.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:47 smithi195 ceph-mon[53157]: pgmap v239: 97 pgs: 12 stale+active+clean, 85 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 407 op/s 2024-04-06T03:28:47.722 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59975]: 2024-04-06 03:28:47.369060662 +0000 UTC m=+0.019497322 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:47.722 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59975]: 2024-04-06 03:28:47.498088495 +0000 UTC m=+0.148525134 container create f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-06T03:28:48.122 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59975]: 2024-04-06 03:28:47.742445951 +0000 UTC m=+0.392882592 container init f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-06T03:28:48.122 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59975]: 2024-04-06 03:28:47.747431856 +0000 UTC m=+0.397868496 container start f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-06T03:28:48.122 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:47 smithi195 podman[59975]: 2024-04-06 03:28:47.850411711 +0000 UTC m=+0.500848351 container attach f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-06T03:28:48.441 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:48 smithi195 podman[59975]: 2024-04-06 03:28:48.123247729 +0000 UTC m=+0.773684373 container died f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:28:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 271401 10.07 MB/sec execute 532 sec latency 32.532 ms 2024-04-06T03:28:48.667 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:48 smithi060 ceph-mon[67116]: pgmap v241: 97 pgs: 2 active+undersized+degraded, 5 peering, 8 stale+active+clean, 82 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 413 op/s; 6/1665 objects degraded (0.360%) 2024-04-06T03:28:48.667 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:48 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 6/1665 objects degraded (0.360%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:48.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:48 smithi195 ceph-mon[53157]: pgmap v241: 97 pgs: 2 active+undersized+degraded, 5 peering, 8 stale+active+clean, 82 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 413 op/s; 6/1665 objects degraded (0.360%) 2024-04-06T03:28:48.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:48 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 6/1665 objects degraded (0.360%), 2 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:49.222 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:48 smithi195 podman[60001]: 2024-04-06 03:28:48.963997774 +0000 UTC m=+0.834918961 container remove f19329734f15c5089ac9b551a5dc47fb63f68d87e6cc4cdaacefd63b6a6f6a6c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:49.223 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:48 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.5.service: Deactivated successfully. 2024-04-06T03:28:49.223 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:48 smithi195 systemd[1]: Stopped Ceph osd.5 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:49.223 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:48 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.5.service: Consumed 5min 41.521s CPU time. 2024-04-06T03:28:49.223 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 systemd[1]: Starting Ceph osd.5 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:49.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 271786 10.08 MB/sec execute 533 sec latency 40.493 ms 2024-04-06T03:28:49.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 podman[60103]: 2024-04-06 03:28:49.333052499 +0000 UTC m=+0.019847962 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:49.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 podman[60103]: 2024-04-06 03:28:49.473495546 +0000 UTC m=+0.160290983 container create 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-06T03:28:49.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 podman[60103]: 2024-04-06 03:28:49.709978999 +0000 UTC m=+0.396774455 container init 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:49.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 podman[60103]: 2024-04-06 03:28:49.714771701 +0000 UTC m=+0.401567139 container start 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-06T03:28:49.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:49 smithi195 podman[60103]: 2024-04-06 03:28:49.826813974 +0000 UTC m=+0.513609421 container attach 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:50.472 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T03:28:50.472 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 bash[60103]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate[60118]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-06T03:28:50.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:50 smithi195 podman[60103]: 2024-04-06 03:28:50.245333072 +0000 UTC m=+0.932128513 container died 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:28:50.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 272169 10.08 MB/sec execute 534 sec latency 41.746 ms 2024-04-06T03:28:50.962 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:50 smithi060 ceph-mon[67116]: pgmap v242: 97 pgs: 8 active+undersized, 18 active+undersized+degraded, 5 peering, 66 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 464 op/s; 135/1647 objects degraded (8.197%) 2024-04-06T03:28:51.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:50 smithi195 ceph-mon[53157]: pgmap v242: 97 pgs: 8 active+undersized, 18 active+undersized+degraded, 5 peering, 66 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 464 op/s; 135/1647 objects degraded (8.197%) 2024-04-06T03:28:51.661 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 podman[60245]: 2024-04-06 03:28:51.290466052 +0000 UTC m=+1.039269036 container remove 0a9bdf50dc733b33d29dbc0b8ac1599529b73faf71d2b2e7aaf33052be4f3fa7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-activate, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:28:51.661 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 podman[60310]: 2024-04-06 03:28:51.413641579 +0000 UTC m=+0.019256262 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:28:51.661 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 podman[60310]: 2024-04-06 03:28:51.524863246 +0000 UTC m=+0.130477968 container create 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235) 2024-04-06T03:28:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 272503 10.07 MB/sec execute 535 sec latency 52.746 ms 2024-04-06T03:28:51.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 podman[60310]: 2024-04-06 03:28:51.803189178 +0000 UTC m=+0.408803860 container init 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-06T03:28:51.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 podman[60310]: 2024-04-06 03:28:51.806603315 +0000 UTC m=+0.412217989 container start 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0) 2024-04-06T03:28:51.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 ceph-osd[60329]: -- 172.21.15.195:0/521270040 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5578e6eb1680 con 0x5578e6e7b000 2024-04-06T03:28:51.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 bash[60310]: 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c 2024-04-06T03:28:51.973 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:51 smithi195 systemd[1]: Started Ceph osd.5 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 272847 10.07 MB/sec execute 536 sec latency 75.799 ms 2024-04-06T03:28:52.722 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:52 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:28:52.351+0000 7f91b0b127c0 -1 Falling back to public interface 2024-04-06T03:28:52.978 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:52 smithi060 ceph-mon[67116]: pgmap v243: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.7 MiB/s wr, 435 op/s; 181/1689 objects degraded (10.716%) 2024-04-06T03:28:52.978 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:52.978 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:52.979 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:53.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:52 smithi195 ceph-mon[53157]: pgmap v243: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.7 MiB/s wr, 435 op/s; 181/1689 objects degraded (10.716%) 2024-04-06T03:28:53.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:53.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:53.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:53.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 273263 10.07 MB/sec execute 537 sec latency 44.898 ms 2024-04-06T03:28:54.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:54 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 196/1737 objects degraded (11.284%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:54.270 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:54 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 196/1737 objects degraded (11.284%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:28:54.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 273582 10.06 MB/sec execute 538 sec latency 46.399 ms 2024-04-06T03:28:55.034 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:55 smithi060 ceph-mon[67116]: pgmap v244: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.9 MiB/s wr, 637 op/s; 196/1737 objects degraded (11.284%) 2024-04-06T03:28:55.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:55 smithi195 ceph-mon[53157]: pgmap v244: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.9 MiB/s wr, 637 op/s; 196/1737 objects degraded (11.284%) 2024-04-06T03:28:55.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 273917 10.06 MB/sec execute 539 sec latency 51.056 ms 2024-04-06T03:28:56.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 274250 10.06 MB/sec execute 540 sec latency 58.385 ms 2024-04-06T03:28:56.953 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:28:57.354 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:57 smithi060 ceph-mon[67116]: pgmap v245: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.5 MiB/s wr, 564 op/s; 195/1698 objects degraded (11.484%) 2024-04-06T03:28:57.354 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:57.354 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:57.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:57 smithi195 ceph-mon[53157]: pgmap v245: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.5 MiB/s wr, 564 op/s; 195/1698 objects degraded (11.484%) 2024-04-06T03:28:57.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:57.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:57.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 274619 10.06 MB/sec execute 541 sec latency 42.247 ms 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (8m) 77s ago 17m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (4m) 77s ago 17m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (4m) 1s ago 16m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 77s ago 17m - - 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (11m) 77s ago 11m 41.1M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (11m) 1s ago 11m 16.3M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (7m) 77s ago 18m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (6m) 1s ago 16m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (5m) 77s ago 19m 51.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (5m) 1s ago 16m 47.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:28:58.061 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (11m) 77s ago 11m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (11m) 1s ago 11m 83.6M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (7m) 77s ago 17m 9470k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (7m) 1s ago 16m 22.0M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (3m) 77s ago 16m 332M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (3m) 77s ago 15m 292M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (2m) 77s ago 15m 214M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (86s) 77s ago 14m 14.8M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (44s) 1s ago 14m 267M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (6s) 1s ago 13m 12.1M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (13m) 1s ago 13m 588M 2704M 17.2.0 e1d6a67b021e 29107db98574 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (13m) 1s ago 13m 640M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:28:58.062 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (6m) 77s ago 16m 63.2M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:28:58.382 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:58 smithi060 ceph-mon[67116]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:58.382 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.383 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:58.383 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.383 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-mon[53157]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:58.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:58.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:28:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 274949 10.06 MB/sec execute 542 sec latency 39.004 ms 2024-04-06T03:28:58.745 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:28:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T03:28:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 9 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:28:58.747 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:28:58.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:58 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:28:58.542+0000 7f91b0b127c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-06T03:28:59.285 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "12/23 daemons upgraded", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:28:59.286 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:28:59.399 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:59 smithi060 ceph-mon[67116]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:59.399 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:59 smithi060 ceph-mon[67116]: pgmap v246: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.0 MiB/s wr, 524 op/s; 188/1659 objects degraded (11.332%) 2024-04-06T03:28:59.399 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:59 smithi060 ceph-mon[67116]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:59.400 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:28:59 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1550580657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:59.623 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:59 smithi195 ceph-mon[53157]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:59.623 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:59 smithi195 ceph-mon[53157]: pgmap v246: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.0 MiB/s wr, 524 op/s; 188/1659 objects degraded (11.332%) 2024-04-06T03:28:59.623 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:59 smithi195 ceph-mon[53157]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:28:59.623 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:28:59 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1550580657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:28:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 275275 10.07 MB/sec execute 543 sec latency 48.082 ms 2024-04-06T03:28:59.944 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 188/1659 objects degraded (11.332%), 22 pgs degraded 2024-04-06T03:28:59.944 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:28:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:28:59.944 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: osd.5 (root=default,host=smithi195) is down 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 188/1659 objects degraded (11.332%), 22 pgs degraded 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-04-06T03:28:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-04-06T03:28:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-04-06T03:29:00.193 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:28:59 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:28:59.926+0000 7f91b0b127c0 -1 osd.5 133 log_to_monitors true 2024-04-06T03:29:00.410 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:00 smithi060 ceph-mon[67116]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:00.410 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:00 smithi060 ceph-mon[67116]: from='osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:29:00.410 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:00 smithi060 ceph-mon[67116]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:29:00.411 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:00 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/650434608' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:29:00.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:00 smithi195 ceph-mon[53157]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:00.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:00 smithi195 ceph-mon[53157]: from='osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:29:00.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:00 smithi195 ceph-mon[53157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:29:00.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:00 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/650434608' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:29:00.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 275555 10.06 MB/sec execute 544 sec latency 38.518 ms 2024-04-06T03:29:00.972 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:29:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:29:00.677+0000 7f919df62700 -1 osd.5 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:29:01.418 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: pgmap v247: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 512 op/s; 180/1635 objects degraded (11.009%) 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 180/1635 objects degraded (11.009%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: osdmap e136: 8 total, 7 up, 8 in 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: from='osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:01.419 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:01 smithi060 ceph-mon[67116]: from='osd.5 ' entity='osd.5' 2024-04-06T03:29:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 275826 10.06 MB/sec execute 545 sec latency 63.977 ms 2024-04-06T03:29:01.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: pgmap v247: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 512 op/s; 180/1635 objects degraded (11.009%) 2024-04-06T03:29:01.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 180/1635 objects degraded (11.009%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:01.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:29:01.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: osdmap e136: 8 total, 7 up, 8 in 2024-04-06T03:29:01.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: from='osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:01.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:01.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:01 smithi195 ceph-mon[53157]: from='osd.5 ' entity='osd.5' 2024-04-06T03:29:02.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: pgmap v249: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 638 op/s; 191/1686 objects degraded (11.329%) 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: pgmap v250: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.2 MiB/s wr, 522 op/s; 191/1686 objects degraded (11.329%) 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:29:02.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865] boot 2024-04-06T03:29:02.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: osdmap e137: 8 total, 8 up, 8 in 2024-04-06T03:29:02.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:02 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:29:02.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 276005 10.04 MB/sec execute 546 sec latency 918.740 ms 2024-04-06T03:29:02.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: pgmap v249: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 638 op/s; 191/1686 objects degraded (11.329%) 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: pgmap v250: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.2 MiB/s wr, 522 op/s; 191/1686 objects degraded (11.329%) 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:29:02.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: osd.5 [v2:172.21.15.195:6808/4227889865,v1:172.21.15.195:6809/4227889865] boot 2024-04-06T03:29:02.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: osdmap e137: 8 total, 8 up, 8 in 2024-04-06T03:29:02.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:02 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:29:03.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 276153 10.03 MB/sec execute 547 sec latency 1918.798 ms 2024-04-06T03:29:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:03 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:03 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T03:29:03.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:03 smithi195 ceph-mon[53157]: osdmap e138: 8 total, 8 up, 8 in 2024-04-06T03:29:03.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:03 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:03.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:03 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T03:29:03.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:03 smithi060 ceph-mon[67116]: osdmap e138: 8 total, 8 up, 8 in 2024-04-06T03:29:04.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:04 smithi060 ceph-mon[67116]: pgmap v253: 97 pgs: 1 remapped+peering, 7 active+undersized, 21 active+undersized+degraded, 68 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 571 B/s rd, 4.0 MiB/s wr, 490 op/s; 176/1701 objects degraded (10.347%) 2024-04-06T03:29:04.413 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:04 smithi060 ceph-mon[67116]: osdmap e139: 8 total, 8 up, 8 in 2024-04-06T03:29:04.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 276342 10.02 MB/sec execute 548 sec latency 2168.319 ms 2024-04-06T03:29:04.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:04 smithi195 ceph-mon[53157]: pgmap v253: 97 pgs: 1 remapped+peering, 7 active+undersized, 21 active+undersized+degraded, 68 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 571 B/s rd, 4.0 MiB/s wr, 490 op/s; 176/1701 objects degraded (10.347%) 2024-04-06T03:29:04.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:04 smithi195 ceph-mon[53157]: osdmap e139: 8 total, 8 up, 8 in 2024-04-06T03:29:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 276523 10.01 MB/sec execute 549 sec latency 970.431 ms 2024-04-06T03:29:05.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:05 smithi060 ceph-mon[67116]: osdmap e140: 8 total, 8 up, 8 in 2024-04-06T03:29:05.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:05 smithi060 ceph-mon[67116]: pgmap v256: 97 pgs: 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 10 active+undersized+degraded, 79 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 MiB/s wr, 98 op/s; 86/1671 objects degraded (5.147%); 391 KiB/s, 24 objects/s recovering 2024-04-06T03:29:05.734 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:05 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 176/1701 objects degraded (10.347%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:05.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:05 smithi195 ceph-mon[53157]: osdmap e140: 8 total, 8 up, 8 in 2024-04-06T03:29:05.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:05 smithi195 ceph-mon[53157]: pgmap v256: 97 pgs: 1 active+recovering+degraded, 2 remapped+peering, 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 10 active+undersized+degraded, 79 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 MiB/s wr, 98 op/s; 86/1671 objects degraded (5.147%); 391 KiB/s, 24 objects/s recovering 2024-04-06T03:29:05.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:05 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 176/1701 objects degraded (10.347%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:06.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 276875 10.00 MB/sec execute 550 sec latency 1009.755 ms 2024-04-06T03:29:06.770 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:06 smithi060 ceph-mon[67116]: osdmap e141: 8 total, 8 up, 8 in 2024-04-06T03:29:06.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:06 smithi195 ceph-mon[53157]: osdmap e141: 8 total, 8 up, 8 in 2024-04-06T03:29:07.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 277175 10.00 MB/sec execute 551 sec latency 65.335 ms 2024-04-06T03:29:07.784 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:07 smithi060 ceph-mon[67116]: pgmap v258: 97 pgs: 2 peering, 1 active+recovering+degraded, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 351 op/s; 12/1596 objects degraded (0.752%); 1.0 MiB/s, 42 objects/s recovering 2024-04-06T03:29:07.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:07 smithi195 ceph-mon[53157]: pgmap v258: 97 pgs: 2 peering, 1 active+recovering+degraded, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 351 op/s; 12/1596 objects degraded (0.752%); 1.0 MiB/s, 42 objects/s recovering 2024-04-06T03:29:08.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 277570 10.00 MB/sec execute 552 sec latency 44.389 ms 2024-04-06T03:29:09.554 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:09 smithi060 ceph-mon[67116]: pgmap v259: 97 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 326 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.9 MiB/s wr, 319 op/s; 7/1602 objects degraded (0.437%); 1.5 MiB/s, 36 objects/s recovering 2024-04-06T03:29:09.664 INFO:teuthology.orchestra.run.smithi060.stdout: 5 278000 10.00 MB/sec execute 553 sec latency 43.755 ms 2024-04-06T03:29:09.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:09 smithi195 ceph-mon[53157]: pgmap v259: 97 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 326 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.9 MiB/s wr, 319 op/s; 7/1602 objects degraded (0.437%); 1.5 MiB/s, 36 objects/s recovering 2024-04-06T03:29:10.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 278350 10.01 MB/sec execute 554 sec latency 45.155 ms 2024-04-06T03:29:10.816 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:10 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 7/1602 objects degraded (0.437%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:29:10.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:10 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 7/1602 objects degraded (0.437%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:29:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 278637 10.01 MB/sec execute 555 sec latency 47.050 ms 2024-04-06T03:29:11.833 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:11 smithi060 ceph-mon[67116]: pgmap v260: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.0 MiB/s wr, 418 op/s; 1.2 MiB/s, 28 objects/s recovering 2024-04-06T03:29:11.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:11 smithi195 ceph-mon[53157]: pgmap v260: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.0 MiB/s wr, 418 op/s; 1.2 MiB/s, 28 objects/s recovering 2024-04-06T03:29:12.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 278970 10.01 MB/sec execute 556 sec latency 34.888 ms 2024-04-06T03:29:12.839 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:12 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/1602 objects degraded (0.437%), 1 pg degraded) 2024-04-06T03:29:12.839 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:12.839 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:12.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:12 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/1602 objects degraded (0.437%), 1 pg degraded) 2024-04-06T03:29:12.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:12.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:13.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 279334 10.01 MB/sec execute 557 sec latency 48.777 ms 2024-04-06T03:29:13.853 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:13 smithi060 ceph-mon[67116]: pgmap v261: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 358 op/s; 887 KiB/s, 15 objects/s recovering 2024-04-06T03:29:13.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:13 smithi195 ceph-mon[53157]: pgmap v261: 97 pgs: 97 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 358 op/s; 887 KiB/s, 15 objects/s recovering 2024-04-06T03:29:14.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 279618 10.00 MB/sec execute 558 sec latency 34.010 ms 2024-04-06T03:29:15.643 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:15 smithi060 ceph-mon[67116]: pgmap v262: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 546 op/s; 773 KiB/s, 13 objects/s recovering 2024-04-06T03:29:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 279960 10.00 MB/sec execute 559 sec latency 49.992 ms 2024-04-06T03:29:15.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:15 smithi195 ceph-mon[53157]: pgmap v262: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 546 op/s; 773 KiB/s, 13 objects/s recovering 2024-04-06T03:29:16.652 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:16.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 280390 9.99 MB/sec execute 560 sec latency 40.745 ms 2024-04-06T03:29:16.886 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:17.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 280737 9.99 MB/sec execute 561 sec latency 40.205 ms 2024-04-06T03:29:17.744 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:17 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:17.744 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:17 smithi195 ceph-mon[53157]: Upgrade: osd.6 is safe to restart 2024-04-06T03:29:17.744 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:17 smithi195 ceph-mon[53157]: pgmap v263: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.6 MiB/s wr, 479 op/s; 479 KiB/s, 4 objects/s recovering 2024-04-06T03:29:17.848 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:17 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T03:29:17.848 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:17 smithi060 ceph-mon[67116]: Upgrade: osd.6 is safe to restart 2024-04-06T03:29:17.848 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:17 smithi060 ceph-mon[67116]: pgmap v263: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.6 MiB/s wr, 479 op/s; 479 KiB/s, 4 objects/s recovering 2024-04-06T03:29:18.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 281204 10.00 MB/sec execute 562 sec latency 41.656 ms 2024-04-06T03:29:18.868 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:18.868 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:18.868 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:29:18.869 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:19.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:19.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:19.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:29:19.211 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 281626 10.00 MB/sec execute 563 sec latency 39.667 ms 2024-04-06T03:29:19.881 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:19 smithi060 ceph-mon[67116]: Upgrade: Updating osd.6 2024-04-06T03:29:19.881 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:19 smithi060 ceph-mon[67116]: Deploying daemon osd.6 on smithi195 2024-04-06T03:29:19.882 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:19 smithi060 ceph-mon[67116]: pgmap v264: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.8 MiB/s wr, 487 op/s; 352 KiB/s, 2 objects/s recovering 2024-04-06T03:29:20.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:19 smithi195 ceph-mon[53157]: Upgrade: Updating osd.6 2024-04-06T03:29:20.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:19 smithi195 ceph-mon[53157]: Deploying daemon osd.6 on smithi195 2024-04-06T03:29:20.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:19 smithi195 ceph-mon[53157]: pgmap v264: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.8 MiB/s wr, 487 op/s; 352 KiB/s, 2 objects/s recovering 2024-04-06T03:29:20.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 281968 10.01 MB/sec execute 564 sec latency 45.511 ms 2024-04-06T03:29:21.659 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:21 smithi060 ceph-mon[67116]: pgmap v265: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.7 MiB/s wr, 567 op/s; 2.3 KiB/s, 1 objects/s recovering 2024-04-06T03:29:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 282340 10.01 MB/sec execute 565 sec latency 35.130 ms 2024-04-06T03:29:21.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:21 smithi195 ceph-mon[53157]: pgmap v265: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.7 MiB/s wr, 567 op/s; 2.3 KiB/s, 1 objects/s recovering 2024-04-06T03:29:22.421 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:22 smithi195 systemd[1]: Stopping Ceph osd.6 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:29:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 282648 10.01 MB/sec execute 566 sec latency 52.014 ms 2024-04-06T03:29:23.177 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:23 smithi060 ceph-mon[67116]: osd.6 marked itself down and dead 2024-04-06T03:29:23.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:23 smithi195 ceph-mon[53157]: osd.6 marked itself down and dead 2024-04-06T03:29:23.473 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[37359]: 2024-04-06T03:29:23.018+0000 7f5dcc08a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:29:23.473 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[37359]: 2024-04-06T03:29:23.018+0000 7f5dcc08a700 -1 osd.6 141 *** Got signal Terminated *** 2024-04-06T03:29:23.473 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[37359]: 2024-04-06T03:29:23.018+0000 7f5dcc08a700 -1 osd.6 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:29:23.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 282827 10.00 MB/sec execute 567 sec latency 595.676 ms 2024-04-06T03:29:24.090 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:23 smithi195 podman[62088]: 2024-04-06 03:29:23.73214067 +0000 UTC m=+1.128720069 container died 29107db985740874de11db79eba5cc823189ba127282b07ed5c65166998a1f48 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_CLEAN=True, release=754, io.buildah.version=1.19.8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T03:29:24.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:24 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:29:24.343 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:24 smithi060 ceph-mon[67116]: osdmap e142: 8 total, 7 up, 8 in 2024-04-06T03:29:24.343 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:24 smithi060 ceph-mon[67116]: pgmap v267: 97 pgs: 10 stale+active+clean, 87 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 552 op/s 2024-04-06T03:29:24.343 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:24 smithi060 ceph-mon[67116]: osdmap e143: 8 total, 7 up, 8 in 2024-04-06T03:29:24.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:24 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:29:24.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:24 smithi195 ceph-mon[53157]: osdmap e142: 8 total, 7 up, 8 in 2024-04-06T03:29:24.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:24 smithi195 ceph-mon[53157]: pgmap v267: 97 pgs: 10 stale+active+clean, 87 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.4 MiB/s wr, 552 op/s 2024-04-06T03:29:24.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:24 smithi195 ceph-mon[53157]: osdmap e143: 8 total, 7 up, 8 in 2024-04-06T03:29:24.473 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:24 smithi195 podman[62088]: 2024-04-06 03:29:24.251901904 +0000 UTC m=+1.648481312 container cleanup 29107db985740874de11db79eba5cc823189ba127282b07ed5c65166998a1f48 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8) 2024-04-06T03:29:24.473 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:24 smithi195 bash[62088]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T03:29:24.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 282971 9.99 MB/sec execute 568 sec latency 1029.357 ms 2024-04-06T03:29:24.972 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:24 smithi195 podman[62105]: 2024-04-06 03:29:24.505720922 +0000 UTC m=+0.769703186 container remove 29107db985740874de11db79eba5cc823189ba127282b07ed5c65166998a1f48 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-04-06T03:29:25.274 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:24 smithi195 podman[62193]: 2024-04-06 03:29:24.88135019 +0000 UTC m=+0.020496983 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:29:25.274 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:25 smithi195 podman[62193]: 2024-04-06 03:29:25.004583991 +0000 UTC m=+0.143730741 container create 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-06T03:29:25.606 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:25 smithi060 ceph-mon[67116]: pgmap v269: 97 pgs: 1 active+undersized, 4 active+undersized+degraded, 5 peering, 87 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.4 MiB/s wr, 516 op/s; 30/1695 objects degraded (1.770%) 2024-04-06T03:29:25.606 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:25 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 30/1695 objects degraded (1.770%), 4 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:25.623 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:25 smithi195 ceph-mon[53157]: pgmap v269: 97 pgs: 1 active+undersized, 4 active+undersized+degraded, 5 peering, 87 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.4 MiB/s wr, 516 op/s; 30/1695 objects degraded (1.770%) 2024-04-06T03:29:25.624 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:25 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 30/1695 objects degraded (1.770%), 4 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:25.624 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:25 smithi195 podman[62193]: 2024-04-06 03:29:25.274240853 +0000 UTC m=+0.413387616 container init 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-06T03:29:25.624 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:25 smithi195 podman[62193]: 2024-04-06 03:29:25.279361515 +0000 UTC m=+0.418508270 container start 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T03:29:25.624 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:25 smithi195 podman[62193]: 2024-04-06 03:29:25.391400161 +0000 UTC m=+0.530546924 container attach 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:29:25.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 283186 9.98 MB/sec execute 569 sec latency 46.054 ms 2024-04-06T03:29:25.938 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:25 smithi195 podman[62193]: 2024-04-06 03:29:25.624243377 +0000 UTC m=+0.763390142 container died 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-06T03:29:26.524 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 podman[62218]: 2024-04-06 03:29:26.254418121 +0000 UTC m=+0.624306677 container remove 5c05386a8b0b7018afc6f10065c194b1db411e6276645e6b6172c62cdb6d9686 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403) 2024-04-06T03:29:26.524 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.6.service: Deactivated successfully. 2024-04-06T03:29:26.524 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 systemd[1]: Stopped Ceph osd.6 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:29:26.524 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.6.service: Consumed 5min 25.336s CPU time. 2024-04-06T03:29:26.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 283516 9.97 MB/sec execute 570 sec latency 62.922 ms 2024-04-06T03:29:26.797 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 systemd[1]: Starting Ceph osd.6 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:29:26.797 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 podman[62314]: 2024-04-06 03:29:26.672530617 +0000 UTC m=+0.021751536 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:29:27.067 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:26 smithi195 podman[62314]: 2024-04-06 03:29:26.797273061 +0000 UTC m=+0.146493953 container create 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-06T03:29:27.067 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 podman[62314]: 2024-04-06 03:29:27.067395288 +0000 UTC m=+0.416616191 container init 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-06T03:29:27.472 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 podman[62314]: 2024-04-06 03:29:27.073195756 +0000 UTC m=+0.422416652 container start 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:29:27.472 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 podman[62314]: 2024-04-06 03:29:27.175787601 +0000 UTC m=+0.525008502 container attach 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-06T03:29:27.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 283955 9.97 MB/sec execute 571 sec latency 44.393 ms 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 bash[62314]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T03:29:27.973 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T03:29:27.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-06T03:29:27.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T03:29:27.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate[62328]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-06T03:29:27.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:27 smithi195 podman[62314]: 2024-04-06 03:29:27.685435047 +0000 UTC m=+1.034655957 container died 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:29:28.434 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:28.435 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:28 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:28.435 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:28 smithi060 ceph-mon[67116]: pgmap v270: 97 pgs: 5 active+undersized, 14 active+undersized+degraded, 5 peering, 73 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 476 op/s; 110/1653 objects degraded (6.655%) 2024-04-06T03:29:28.632 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:28.632 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:28 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:28.632 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:28 smithi195 ceph-mon[53157]: pgmap v270: 97 pgs: 5 active+undersized, 14 active+undersized+degraded, 5 peering, 73 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 476 op/s; 110/1653 objects degraded (6.655%) 2024-04-06T03:29:28.632 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 podman[62492]: 2024-04-06 03:29:28.36418481 +0000 UTC m=+0.672649946 container remove 322dc469e05f4494818501d3429a170157fbb512f25b365f559dc22c4a0fb49f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:29:28.632 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 podman[62554]: 2024-04-06 03:29:28.505649778 +0000 UTC m=+0.021244159 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:29:28.632 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 podman[62554]: 2024-04-06 03:29:28.632043777 +0000 UTC m=+0.147638165 container create f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-06T03:29:28.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 284277 9.96 MB/sec execute 572 sec latency 34.394 ms 2024-04-06T03:29:29.222 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 podman[62554]: 2024-04-06 03:29:28.902631725 +0000 UTC m=+0.418226116 container init f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235) 2024-04-06T03:29:29.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 podman[62554]: 2024-04-06 03:29:28.906340837 +0000 UTC m=+0.421935218 container start f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:29:29.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:28 smithi195 ceph-osd[62572]: -- 172.21.15.195:0/3338717139 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccad7d3680 con 0x55ccad79d000 2024-04-06T03:29:29.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:29 smithi195 bash[62554]: f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb 2024-04-06T03:29:29.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:29 smithi195 systemd[1]: Started Ceph osd.6 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:29:29.665 INFO:teuthology.orchestra.run.smithi060.stdout: 5 284681 9.97 MB/sec execute 573 sec latency 73.064 ms 2024-04-06T03:29:29.722 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:29:29.450+0000 7fe7818d57c0 -1 Falling back to public interface 2024-04-06T03:29:30.229 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:30 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:30.229 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:30 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:30.230 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:30 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:30.230 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:30 smithi060 ceph-mon[67116]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.2 MiB/s wr, 449 op/s; 193/1644 objects degraded (11.740%) 2024-04-06T03:29:30.408 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:29:30.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:30 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:30.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:30 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:30.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:30 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:30.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:30 smithi195 ceph-mon[53157]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.2 MiB/s wr, 449 op/s; 193/1644 objects degraded (11.740%) 2024-04-06T03:29:30.671 INFO:teuthology.orchestra.run.smithi060.stdout: 5 285111 9.97 MB/sec execute 574 sec latency 42.883 ms 2024-04-06T03:29:31.247 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:31 smithi060 ceph-mon[67116]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:31.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:31 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 193/1644 objects degraded (11.740%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:31.260 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:31 smithi195 ceph-mon[53157]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:31.260 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:31 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 193/1644 objects degraded (11.740%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:31.296 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:29:31.296 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (8m) 111s ago 18m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (5m) 111s ago 17m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (4m) 35s ago 17m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 111s ago 17m - - 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (12m) 111s ago 12m 41.1M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (12m) 35s ago 12m 16.3M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (7m) 111s ago 19m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (6m) 35s ago 17m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (6m) 111s ago 19m 51.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (5m) 35s ago 17m 47.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (11m) 111s ago 11m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (11m) 35s ago 11m 83.6M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (8m) 111s ago 17m 9470k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (7m) 35s ago 17m 22.0M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (4m) 111s ago 16m 332M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:29:31.297 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (3m) 111s ago 16m 292M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (2m) 111s ago 15m 214M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (119s) 111s ago 15m 14.8M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (77s) 35s ago 14m 267M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (39s) 35s ago 14m 12.1M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 starting - - - 2704M 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (13m) 35s ago 13m 640M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:29:31.298 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (6m) 111s ago 16m 63.2M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:29:31.671 INFO:teuthology.orchestra.run.smithi060.stdout: 5 285485 9.98 MB/sec execute 575 sec latency 36.978 ms 2024-04-06T03:29:31.815 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:29:31.815 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 6 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-06T03:29:31.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 10 2024-04-06T03:29:31.817 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:29:31.817 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:29:32.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:32 smithi195 ceph-mon[53157]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:32.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:32 smithi195 ceph-mon[53157]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:32.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:32 smithi195 ceph-mon[53157]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.9 MiB/s wr, 482 op/s; 199/1671 objects degraded (11.909%) 2024-04-06T03:29:32.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:32 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/4066399730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "12/23 daemons upgraded", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:29:32.227 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:29:32.264 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:32 smithi060 ceph-mon[67116]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:32.264 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:32 smithi060 ceph-mon[67116]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:32.264 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:32 smithi060 ceph-mon[67116]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.9 MiB/s wr, 482 op/s; 199/1671 objects degraded (11.909%) 2024-04-06T03:29:32.264 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:32 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/4066399730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:32.671 INFO:teuthology.orchestra.run.smithi060.stdout: 5 285759 9.98 MB/sec execute 576 sec latency 50.523 ms 2024-04-06T03:29:32.783 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 199/1671 objects degraded (11.909%), 25 pgs degraded 2024-04-06T03:29:32.783 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:29:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: osd.6 (root=default,host=smithi195) is down 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 199/1671 objects degraded (11.909%), 25 pgs degraded 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-04-06T03:29:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-04-06T03:29:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-04-06T03:29:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-04-06T03:29:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-04-06T03:29:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-04-06T03:29:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-04-06T03:29:33.295 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:33 smithi060 ceph-mon[67116]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:33.296 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:33 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3066051121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:29:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:33 smithi195 ceph-mon[53157]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:29:33.496 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:33 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3066051121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 286073 9.98 MB/sec execute 577 sec latency 49.096 ms 2024-04-06T03:29:34.579 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:34 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:34.579 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:34 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:34.580 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:34 smithi060 ceph-mon[67116]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 566 op/s; 200/1668 objects degraded (11.990%) 2024-04-06T03:29:34.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 286363 9.97 MB/sec execute 578 sec latency 54.357 ms 2024-04-06T03:29:34.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:34 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:34.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:34 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:34.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:34 smithi195 ceph-mon[53157]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 566 op/s; 200/1668 objects degraded (11.990%) 2024-04-06T03:29:35.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 286594 9.97 MB/sec execute 579 sec latency 62.539 ms 2024-04-06T03:29:35.719 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:35 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:29:35.423+0000 7fe7818d57c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-06T03:29:35.884 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:35.884 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:35.885 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:35 smithi060 ceph-mon[67116]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.0 MiB/s wr, 515 op/s; 207/1695 objects degraded (12.212%) 2024-04-06T03:29:35.885 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:35 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 200/1668 objects degraded (11.990%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:35.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:35.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:35.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:35 smithi195 ceph-mon[53157]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.0 MiB/s wr, 515 op/s; 207/1695 objects degraded (12.212%) 2024-04-06T03:29:35.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:35 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 200/1668 objects degraded (11.990%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:36.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 286902 9.97 MB/sec execute 580 sec latency 71.174 ms 2024-04-06T03:29:37.080 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:36 smithi195 ceph-mon[53157]: from='osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:29:37.081 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:36 smithi195 ceph-mon[53157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:29:37.081 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:36 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:29:36.902+0000 7fe7818d57c0 -1 osd.6 141 log_to_monitors true 2024-04-06T03:29:37.081 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:29:37.006+0000 7fe772d2d700 -1 osd.6 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:29:37.149 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:36 smithi060 ceph-mon[67116]: from='osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:29:37.149 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:36 smithi060 ceph-mon[67116]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:29:37.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 287323 9.96 MB/sec execute 581 sec latency 41.961 ms 2024-04-06T03:29:38.133 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-mon[53157]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:29:38.134 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-mon[53157]: osdmap e144: 8 total, 7 up, 8 in 2024-04-06T03:29:38.134 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-mon[53157]: from='osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:38.134 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-mon[53157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:38.134 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:37 smithi195 ceph-mon[53157]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.1 MiB/s wr, 527 op/s; 212/1719 objects degraded (12.333%) 2024-04-06T03:29:38.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:37 smithi060 ceph-mon[67116]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:29:38.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:37 smithi060 ceph-mon[67116]: osdmap e144: 8 total, 7 up, 8 in 2024-04-06T03:29:38.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:37 smithi060 ceph-mon[67116]: from='osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:38.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:37 smithi060 ceph-mon[67116]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:29:38.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:37 smithi060 ceph-mon[67116]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.1 MiB/s wr, 527 op/s; 212/1719 objects degraded (12.333%) 2024-04-06T03:29:38.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 287469 9.94 MB/sec execute 582 sec latency 701.310 ms 2024-04-06T03:29:39.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839] boot 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: osdmap e145: 8 total, 8 up, 8 in 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:29:39.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: osd.6 [v2:172.21.15.195:6816/2717336839,v1:172.21.15.195:6817/2717336839] boot 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: osdmap e145: 8 total, 8 up, 8 in 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:39.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 287469 9.92 MB/sec execute 583 sec latency 1701.344 ms 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: pgmap v278: 97 pgs: 4 peering, 10 active+undersized, 21 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.9 MiB/s wr, 501 op/s; 169/1722 objects degraded (9.814%) 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: pgmap v279: 97 pgs: 4 peering, 10 active+undersized, 21 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.9 MiB/s wr, 384 op/s; 169/1722 objects degraded (9.814%) 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: osdmap e146: 8 total, 8 up, 8 in 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:40.236 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:39 smithi060 ceph-mon[67116]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T03:29:40.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: pgmap v278: 97 pgs: 4 peering, 10 active+undersized, 21 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.9 MiB/s wr, 501 op/s; 169/1722 objects degraded (9.814%) 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: pgmap v279: 97 pgs: 4 peering, 10 active+undersized, 21 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.9 MiB/s wr, 384 op/s; 169/1722 objects degraded (9.814%) 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: osdmap e146: 8 total, 8 up, 8 in 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:40.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:39 smithi195 ceph-mon[53157]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T03:29:40.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 287689 9.91 MB/sec execute 584 sec latency 2084.731 ms 2024-04-06T03:29:40.994 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:40 smithi060 ceph-mon[67116]: osdmap e147: 8 total, 8 up, 8 in 2024-04-06T03:29:40.995 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:40 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 169/1722 objects degraded (9.814%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:40.995 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:40 smithi060 ceph-mon[67116]: osdmap e148: 8 total, 8 up, 8 in 2024-04-06T03:29:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:40 smithi195 ceph-mon[53157]: osdmap e147: 8 total, 8 up, 8 in 2024-04-06T03:29:41.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:40 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 169/1722 objects degraded (9.814%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T03:29:41.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:40 smithi195 ceph-mon[53157]: osdmap e148: 8 total, 8 up, 8 in 2024-04-06T03:29:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 287984 9.91 MB/sec execute 585 sec latency 935.969 ms 2024-04-06T03:29:42.017 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:41 smithi060 ceph-mon[67116]: pgmap v283: 97 pgs: 1 remapped+peering, 4 peering, 8 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 67 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s wr, 138 op/s; 126/1599 objects degraded (7.880%); 58 KiB/s, 77 keys/s, 7 objects/s recovering 2024-04-06T03:29:42.017 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:41 smithi060 ceph-mon[67116]: osdmap e149: 8 total, 8 up, 8 in 2024-04-06T03:29:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:41 smithi195 ceph-mon[53157]: pgmap v283: 97 pgs: 1 remapped+peering, 4 peering, 8 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 67 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s wr, 138 op/s; 126/1599 objects degraded (7.880%); 58 KiB/s, 77 keys/s, 7 objects/s recovering 2024-04-06T03:29:42.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:41 smithi195 ceph-mon[53157]: osdmap e149: 8 total, 8 up, 8 in 2024-04-06T03:29:42.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 288419 9.92 MB/sec execute 586 sec latency 947.844 ms 2024-04-06T03:29:43.554 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:43.555 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 288822 9.92 MB/sec execute 587 sec latency 36.355 ms 2024-04-06T03:29:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:43.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:44.567 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:44 smithi060 ceph-mon[67116]: pgmap v285: 97 pgs: 1 remapped+peering, 5 peering, 5 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+undersized+degraded, 82 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.4 MiB/s wr, 372 op/s; 37/1572 objects degraded (2.354%); 4.3 MiB/s, 56 keys/s, 23 objects/s recovering 2024-04-06T03:29:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 289219 9.94 MB/sec execute 588 sec latency 38.619 ms 2024-04-06T03:29:44.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:44 smithi195 ceph-mon[53157]: pgmap v285: 97 pgs: 1 remapped+peering, 5 peering, 5 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+undersized+degraded, 82 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 2.4 MiB/s wr, 372 op/s; 37/1572 objects degraded (2.354%); 4.3 MiB/s, 56 keys/s, 23 objects/s recovering 2024-04-06T03:29:45.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 289494 9.93 MB/sec execute 589 sec latency 77.391 ms 2024-04-06T03:29:46.395 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:46 smithi060 ceph-mon[67116]: pgmap v286: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 94 active+clean; 324 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.5 MiB/s wr, 331 op/s; 19/1581 objects degraded (1.202%); 3.1 MiB/s, 38 keys/s, 31 objects/s recovering 2024-04-06T03:29:46.395 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:46 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 19/1581 objects degraded (1.202%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:29:46.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 289813 9.93 MB/sec execute 590 sec latency 37.888 ms 2024-04-06T03:29:46.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:46 smithi195 ceph-mon[53157]: pgmap v286: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 94 active+clean; 324 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.5 MiB/s wr, 331 op/s; 19/1581 objects degraded (1.202%); 3.1 MiB/s, 38 keys/s, 31 objects/s recovering 2024-04-06T03:29:46.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:46 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 19/1581 objects degraded (1.202%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:29:47.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 290071 9.92 MB/sec execute 591 sec latency 52.299 ms 2024-04-06T03:29:47.685 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:47 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/1581 objects degraded (1.202%), 1 pg degraded) 2024-04-06T03:29:47.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:47 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/1581 objects degraded (1.202%), 1 pg degraded) 2024-04-06T03:29:48.454 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:48 smithi060 ceph-mon[67116]: pgmap v287: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 491 op/s; 2.6 MiB/s, 29 objects/s recovering 2024-04-06T03:29:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: 5 290255 9.91 MB/sec execute 592 sec latency 44.764 ms 2024-04-06T03:29:48.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:48 smithi195 ceph-mon[53157]: pgmap v287: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 491 op/s; 2.6 MiB/s, 29 objects/s recovering 2024-04-06T03:29:49.483 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:49 smithi060 ceph-mon[67116]: pgmap v288: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.8 MiB/s wr, 442 op/s; 2.2 MiB/s, 22 objects/s recovering 2024-04-06T03:29:49.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 290506 9.90 MB/sec execute 593 sec latency 62.808 ms 2024-04-06T03:29:49.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:49 smithi195 ceph-mon[53157]: pgmap v288: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.8 MiB/s wr, 442 op/s; 2.2 MiB/s, 22 objects/s recovering 2024-04-06T03:29:50.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 290832 9.90 MB/sec execute 594 sec latency 64.046 ms 2024-04-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 291184 9.89 MB/sec execute 595 sec latency 56.896 ms 2024-04-06T03:29:52.062 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:51 smithi060 ceph-mon[67116]: pgmap v289: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 456 op/s; 1.8 MiB/s, 18 objects/s recovering 2024-04-06T03:29:52.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:51 smithi195 ceph-mon[53157]: pgmap v289: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 456 op/s; 1.8 MiB/s, 18 objects/s recovering 2024-04-06T03:29:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 291499 9.89 MB/sec execute 596 sec latency 78.191 ms 2024-04-06T03:29:53.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 291861 9.89 MB/sec execute 597 sec latency 39.211 ms 2024-04-06T03:29:53.890 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:53.890 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:53 smithi060 ceph-mon[67116]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.8 MiB/s wr, 505 op/s; 1.1 MiB/s, 13 objects/s recovering 2024-04-06T03:29:54.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:54.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:53 smithi195 ceph-mon[53157]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.8 MiB/s wr, 505 op/s; 1.1 MiB/s, 13 objects/s recovering 2024-04-06T03:29:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 292203 9.89 MB/sec execute 598 sec latency 52.183 ms 2024-04-06T03:29:55.033 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:54 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:55.033 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:54 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:55.034 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:54 smithi060 ceph-mon[67116]: Upgrade: osd.7 is safe to restart 2024-04-06T03:29:55.089 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:54 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:55.089 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:54 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T03:29:55.089 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:54 smithi195 ceph-mon[53157]: Upgrade: osd.7 is safe to restart 2024-04-06T03:29:55.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 292553 9.90 MB/sec execute 599 sec latency 39.857 ms 2024-04-06T03:29:55.811 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:55 smithi060 ceph-mon[67116]: pgmap v291: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 443 op/s; 79 KiB/s, 9 objects/s recovering 2024-04-06T03:29:56.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:55 smithi195 ceph-mon[53157]: pgmap v291: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 443 op/s; 79 KiB/s, 9 objects/s recovering 2024-04-06T03:29:56.672 INFO:teuthology.orchestra.run.smithi060.stdout: 5 cleanup 600 sec 2024-04-06T03:29:56.949 INFO:teuthology.orchestra.run.smithi060.stdout: 0 cleanup 600 sec 2024-04-06T03:29:56.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:29:56.949 INFO:teuthology.orchestra.run.smithi060.stdout: Operation Count AvgLat MaxLat 2024-04-06T03:29:56.949 INFO:teuthology.orchestra.run.smithi060.stdout: ---------------------------------------- 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: NTCreateX 190045 4.628 2158.199 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Close 139633 0.143 8.137 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Rename 8066 9.934 83.300 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Unlink 38344 6.803 2072.345 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Qpathinfo 172495 1.175 2140.690 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Qfileinfo 30122 0.008 1.472 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Qfsinfo 31628 0.632 17.294 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Sfileinfo 15479 12.739 2085.610 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: Find 66692 1.031 16.686 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: WriteX 94025 13.442 2319.436 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: ReadX 298922 0.010 3.281 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: LockX 624 0.433 3.659 2024-04-06T03:29:56.950 INFO:teuthology.orchestra.run.smithi060.stdout: UnlockX 624 0.375 3.089 2024-04-06T03:29:56.951 INFO:teuthology.orchestra.run.smithi060.stdout: Flush 13305 0.002 0.023 2024-04-06T03:29:56.951 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:29:56.951 INFO:teuthology.orchestra.run.smithi060.stdout:Throughput 9.89656 MB/sec 5 clients 5 procs max_latency=2319.441 ms 2024-04-06T03:29:56.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-04-06T03:29:57.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-04-06T03:29:57.149 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-04-06T03:29:57.187 INFO:teuthology.orchestra.run.smithi060.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-06T03:29:57.187 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:29:57.187 INFO:teuthology.orchestra.run.smithi060.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-04-06T03:29:57.192 INFO:teuthology.orchestra.run.smithi060.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-06T03:29:57.288 INFO:teuthology.orchestra.run.smithi060.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-06T03:29:57.288 INFO:teuthology.orchestra.run.smithi060.stdout:releasing clients 2024-04-06T03:29:57.341 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:57 smithi060 ceph-mon[67116]: Upgrade: Updating osd.7 2024-04-06T03:29:57.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:57.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:29:57.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:57.342 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:57 smithi060 ceph-mon[67116]: Deploying daemon osd.7 on smithi195 2024-04-06T03:29:57.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:57 smithi195 ceph-mon[53157]: Upgrade: Updating osd.7 2024-04-06T03:29:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:29:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:57 smithi195 ceph-mon[53157]: Deploying daemon osd.7 on smithi195 2024-04-06T03:29:58.601 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:58 smithi195 ceph-mon[53157]: pgmap v292: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 433 op/s; 12 KiB/s, 1 objects/s recovering 2024-04-06T03:29:58.601 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:58.601 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:29:58 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:58.625 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:58 smithi060 ceph-mon[67116]: pgmap v292: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 433 op/s; 12 KiB/s, 1 objects/s recovering 2024-04-06T03:29:58.625 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:29:58.625 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:29:58 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: 5 342 23.41 MB/sec execute 1 sec latency 92.326 ms 2024-04-06T03:30:00.288 INFO:teuthology.orchestra.run.smithi060.stdout: 5 455 20.94 MB/sec execute 2 sec latency 76.005 ms 2024-04-06T03:30:00.475 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:00 smithi060 ceph-mon[67116]: pgmap v293: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 436 op/s 2024-04-06T03:30:00.476 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:00 smithi060 ceph-mon[67116]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:30:00.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:00 smithi195 ceph-mon[53157]: pgmap v293: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 436 op/s 2024-04-06T03:30:00.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:00 smithi195 ceph-mon[53157]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:30:00.722 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:00 smithi195 systemd[1]: Stopping Ceph osd.7 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:30:00.972 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[40442]: 2024-04-06T03:30:00.720+0000 7f6daa8b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:30:00.973 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[40442]: 2024-04-06T03:30:00.720+0000 7f6daa8b9700 -1 osd.7 149 *** Got signal Terminated *** 2024-04-06T03:30:00.973 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[40442]: 2024-04-06T03:30:00.720+0000 7f6daa8b9700 -1 osd.7 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:30:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: 5 674 22.56 MB/sec execute 3 sec latency 116.904 ms 2024-04-06T03:30:01.572 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:01 smithi060 ceph-mon[67116]: osd.7 marked itself down and dead 2024-04-06T03:30:01.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:01 smithi195 ceph-mon[53157]: osd.7 marked itself down and dead 2024-04-06T03:30:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: 5 757 17.12 MB/sec execute 4 sec latency 894.188 ms 2024-04-06T03:30:02.379 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:02 smithi195 podman[64345]: 2024-04-06 03:30:02.084658556 +0000 UTC m=+1.507084358 container died 79907b371d9acaea3126fafb0975332a47af50807c359e34d6d6ae299b7e1b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=, name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T03:30:02.630 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:02 smithi060 ceph-mon[67116]: pgmap v294: 97 pgs: 97 active+clean; 363 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 7.9 MiB/s wr, 440 op/s 2024-04-06T03:30:02.630 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:02 smithi060 ceph-mon[67116]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:30:02.630 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:02 smithi060 ceph-mon[67116]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-06T03:30:02.630 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:02 smithi060 ceph-mon[67116]: osdmap e150: 8 total, 7 up, 8 in 2024-04-06T03:30:02.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:02 smithi195 ceph-mon[53157]: pgmap v294: 97 pgs: 97 active+clean; 363 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 7.9 MiB/s wr, 440 op/s 2024-04-06T03:30:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:02 smithi195 ceph-mon[53157]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T03:30:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:02 smithi195 ceph-mon[53157]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-06T03:30:02.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:02 smithi195 ceph-mon[53157]: osdmap e150: 8 total, 7 up, 8 in 2024-04-06T03:30:02.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:02 smithi195 podman[64345]: 2024-04-06 03:30:02.612026739 +0000 UTC m=+2.034452535 container cleanup 79907b371d9acaea3126fafb0975332a47af50807c359e34d6d6ae299b7e1b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T03:30:02.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:02 smithi195 bash[64345]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T03:30:03.223 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:02 smithi195 podman[64366]: 2024-04-06 03:30:02.822280283 +0000 UTC m=+0.738073682 container remove 79907b371d9acaea3126fafb0975332a47af50807c359e34d6d6ae299b7e1b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, name=centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-04-06T03:30:03.258 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:30:03.288 INFO:teuthology.orchestra.run.smithi060.stdout: 5 cleanup 5 sec 2024-04-06T03:30:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: 0 cleanup 5 sec 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Operation Count AvgLat MaxLat 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: ---------------------------------------- 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: NTCreateX 599 9.606 75.999 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Close 556 11.383 1048.525 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Rename 23 19.674 33.817 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Unlink 49 2.554 16.020 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Qpathinfo 183 1.151 16.975 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Qfileinfo 323 26.515 1012.832 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Qfsinfo 101 0.416 1.998 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Sfileinfo 92 11.948 27.011 2024-04-06T03:30:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: Find 132 0.777 1.299 2024-04-06T03:30:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: WriteX 1570 0.025 0.066 2024-04-06T03:30:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: ReadX 458 0.019 0.522 2024-04-06T03:30:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: Flush 115 20.744 1059.003 2024-04-06T03:30:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:30:03.528 INFO:teuthology.orchestra.run.smithi060.stdout:Throughput 17.1177 MB/sec 5 clients 5 procs max_latency=894.188 ms 2024-04-06T03:30:03.529 DEBUG:teuthology.parallel:result is None 2024-04-06T03:30:03.602 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:03 smithi195 ceph-mon[53157]: osdmap e151: 8 total, 7 up, 8 in 2024-04-06T03:30:03.602 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.212173781 +0000 UTC m=+0.023045814 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:30:03.602 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.348650541 +0000 UTC m=+0.159522548 container create 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-06T03:30:03.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:03 smithi060 ceph-mon[67116]: osdmap e151: 8 total, 7 up, 8 in 2024-04-06T03:30:03.894 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.601865311 +0000 UTC m=+0.412737325 container init 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-06T03:30:03.895 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.605640404 +0000 UTC m=+0.416512418 container start 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:30:03.895 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.693408056 +0000 UTC m=+0.504280070 container attach 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-06T03:30:03.895 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:03 smithi195 podman[64451]: 2024-04-06 03:30:03.89500118 +0000 UTC m=+0.705873197 container died 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (9m) 2m ago 18m 17.6M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (5m) 2m ago 18m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (5m) 30s ago 17m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 2m ago 18m - - 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (12m) 2m ago 12m 41.1M - 17.2.0 e1d6a67b021e a63f1751fdf1 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (12m) 30s ago 12m 16.4M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:30:04.027 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (8m) 2m ago 19m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (7m) 30s ago 17m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (7m) 2m ago 20m 51.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (6m) 30s ago 17m 46.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (12m) 2m ago 12m 282M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (12m) 30s ago 12m 83.7M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (8m) 2m ago 18m 9470k - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (8m) 30s ago 17m 17.6M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (4m) 2m ago 17m 332M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (4m) 2m ago 16m 292M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (3m) 2m ago 16m 214M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (2m) 2m ago 15m 14.8M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (110s) 30s ago 15m 309M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (72s) 30s ago 14m 263M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:30:04.028 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (35s) 30s ago 14m 14.8M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:30:04.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (14m) 30s ago 14m 687M 2704M 17.2.0 e1d6a67b021e 79907b371d9a 2024-04-06T03:30:04.029 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (7m) 2m ago 17m 63.2M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:30:04.434 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:30:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:30:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:30:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 7 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 11 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:30:04.435 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:30:04.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:04 smithi195 ceph-mon[53157]: pgmap v297: 97 pgs: 6 peering, 12 stale+active+clean, 79 active+clean; 358 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 485 op/s 2024-04-06T03:30:04.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:04 smithi195 ceph-mon[53157]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:04.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 podman[64477]: 2024-04-06 03:30:04.488618267 +0000 UTC m=+0.587843735 container remove 16f82a29243cf3ba6f55929dc9416464d480116e95afeeaed11ac2f85100b639 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-06T03:30:04.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.7.service: Deactivated successfully. 2024-04-06T03:30:04.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 systemd[1]: Stopped Ceph osd.7 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:30:04.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.7.service: Consumed 8min 543ms CPU time. 2024-04-06T03:30:04.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:04 smithi060 ceph-mon[67116]: pgmap v297: 97 pgs: 6 peering, 12 stale+active+clean, 79 active+clean; 358 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 485 op/s 2024-04-06T03:30:04.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:04 smithi060 ceph-mon[67116]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:04.779 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:30:04.779 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:30:04.779 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:30:04.779 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "13/23 daemons upgraded", 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:30:04.780 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:30:05.157 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 systemd[1]: Starting Ceph osd.7 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:30:05.157 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:04 smithi195 podman[64577]: 2024-04-06 03:30:04.844722387 +0000 UTC m=+0.018400506 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:30:05.157 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:05 smithi195 podman[64577]: 2024-04-06 03:30:05.157078297 +0000 UTC m=+0.330756408 container create 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0) 2024-04-06T03:30:05.187 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-04-06T03:30:05.187 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:30:05.188 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:30:05.188 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T03:30:05.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd.7 (root=default,host=smithi195) is down 2024-04-06T03:30:05.188 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-06T03:30:05.188 INFO:teuthology.orchestra.run.smithi060.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-06T03:30:05.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:05 smithi060 ceph-mon[67116]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:05.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:05 smithi060 ceph-mon[67116]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:05.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:05 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1389652106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:05.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:05 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1056503592' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:30:05.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:05 smithi195 ceph-mon[53157]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:05.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:05 smithi195 ceph-mon[53157]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:05.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:05 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1389652106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:05.699 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:05 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1056503592' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:30:05.972 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:05 smithi195 podman[64577]: 2024-04-06 03:30:05.835393123 +0000 UTC m=+1.009071243 container init 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-06T03:30:05.973 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:05 smithi195 podman[64577]: 2024-04-06 03:30:05.83909721 +0000 UTC m=+1.012775330 container start 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-06T03:30:05.973 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:05 smithi195 podman[64577]: 2024-04-06 03:30:05.910100061 +0000 UTC m=+1.083778175 container attach 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-06T03:30:06.672 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-mon[53157]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:06.672 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-mon[53157]: pgmap v298: 97 pgs: 3 active+undersized, 17 active+undersized+degraded, 6 peering, 3 stale+active+clean, 68 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 15 MiB/s wr, 505 op/s; 103/1347 objects degraded (7.647%) 2024-04-06T03:30:06.672 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-mon[53157]: Health check failed: Degraded data redundancy: 103/1347 objects degraded (7.647%), 17 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 bash[64577]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T03:30:06.673 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate[64592]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-06T03:30:06.674 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 podman[64577]: 2024-04-06 03:30:06.307036961 +0000 UTC m=+1.480715075 container died 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:30:06.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:06 smithi060 ceph-mon[67116]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:06.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:06 smithi060 ceph-mon[67116]: pgmap v298: 97 pgs: 3 active+undersized, 17 active+undersized+degraded, 6 peering, 3 stale+active+clean, 68 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 15 MiB/s wr, 505 op/s; 103/1347 objects degraded (7.647%) 2024-04-06T03:30:06.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:06 smithi060 ceph-mon[67116]: Health check failed: Degraded data redundancy: 103/1347 objects degraded (7.647%), 17 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:07.222 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:06 smithi195 podman[64709]: 2024-04-06 03:30:06.969021873 +0000 UTC m=+0.653130337 container remove 116a08885a5113d2f43675f189967d0b3b5b106fce5a9387f2b79f1d241d9a94 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-06T03:30:07.223 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 podman[64768]: 2024-04-06 03:30:07.083573188 +0000 UTC m=+0.018684869 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:30:07.223 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 podman[64768]: 2024-04-06 03:30:07.206857909 +0000 UTC m=+0.141969589 container create e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-06T03:30:07.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 podman[64768]: 2024-04-06 03:30:07.44319643 +0000 UTC m=+0.378308121 container init e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-06T03:30:07.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 podman[64768]: 2024-04-06 03:30:07.446623137 +0000 UTC m=+0.381734818 container start e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-06T03:30:07.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 ceph-osd[64819]: -- 172.21.15.195:0/3284202918 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9afb65680 con 0x55d9afb2f000 2024-04-06T03:30:07.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 bash[64768]: e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 2024-04-06T03:30:07.723 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 systemd[1]: Started Ceph osd.7 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:30:08.422 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:30:07.994+0000 7f80c10e87c0 -1 Falling back to public interface 2024-04-06T03:30:08.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:08 smithi195 ceph-mon[53157]: pgmap v299: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 5 peering, 55 active+clean; 307 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 329 op/s; 145/1146 objects degraded (12.653%) 2024-04-06T03:30:08.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:08 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:08.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:08 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:08.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:08 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:08.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:08 smithi060 ceph-mon[67116]: pgmap v299: 97 pgs: 8 active+undersized, 29 active+undersized+degraded, 5 peering, 55 active+clean; 307 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s wr, 329 op/s; 145/1146 objects degraded (12.653%) 2024-04-06T03:30:08.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:08 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:08.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:08 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:08.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:08 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:09.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:09 smithi195 ceph-mon[53157]: pgmap v300: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 279 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 MiB/s wr, 332 op/s; 106/897 objects degraded (11.817%) 2024-04-06T03:30:09.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:09 smithi060 ceph-mon[67116]: pgmap v300: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 279 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 MiB/s wr, 332 op/s; 106/897 objects degraded (11.817%) 2024-04-06T03:30:10.972 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:10 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 106/897 objects degraded (11.817%), 33 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:10.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:10 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 106/897 objects degraded (11.817%), 33 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:12.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:12 smithi195 ceph-mon[53157]: pgmap v301: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 225 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.2 MiB/s wr, 273 op/s; 71/435 objects degraded (16.322%) 2024-04-06T03:30:12.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:12.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:12.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:12 smithi060 ceph-mon[67116]: pgmap v301: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 225 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.2 MiB/s wr, 273 op/s; 71/435 objects degraded (16.322%) 2024-04-06T03:30:12.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:12.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:13 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:13.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:13.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:13 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:14.381 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:14 smithi195 ceph-mon[53157]: pgmap v302: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 129 op/s; 70/429 objects degraded (16.317%) 2024-04-06T03:30:14.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:14 smithi060 ceph-mon[67116]: pgmap v302: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 129 op/s; 70/429 objects degraded (16.317%) 2024-04-06T03:30:14.973 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:30:14.672+0000 7f80c10e87c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-06T03:30:16.378 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:30:16.252+0000 7f80c10e87c0 -1 osd.7 149 log_to_monitors true 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: pgmap v303: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s wr, 113 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 69/426 objects degraded (16.197%), 33 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T03:30:16.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-06T03:30:16.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: pgmap v303: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s wr, 113 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 69/426 objects degraded (16.197%), 33 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:16.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T03:30:16.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T03:30:16.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T03:30:16.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T03:30:16.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T03:30:16.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T03:30:16.753 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-06T03:30:16.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-06T03:30:16.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-06T03:30:16.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-06T03:30:16.754 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-06T03:30:17.473 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:30:17.191+0000 7f80b2540700 -1 osd.7 149 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T03:30:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: pgmap v304: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 643 KiB/s wr, 74 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: pgmap v305: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 213 KiB/s wr, 64 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all osd 2024-04-06T03:30:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: Upgrade: Setting require_osd_release to 19 squid 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: osdmap e152: 8 total, 7 up, 8 in 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi060.hrodqu"]}]: dispatch 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: from='osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:30:17.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:17 smithi195 ceph-mon[53157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: pgmap v304: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 643 KiB/s wr, 74 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: pgmap v305: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 213 KiB/s wr, 64 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all osd 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: Upgrade: Setting require_osd_release to 19 squid 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: osdmap e152: 8 total, 7 up, 8 in 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi060.hrodqu"]}]: dispatch 2024-04-06T03:30:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: from='osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:30:17.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:17 smithi060 ceph-mon[67116]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-06T03:30:18.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:18 smithi195 ceph-mon[53157]: Upgrade: It appears safe to stop mds.foofs.smithi060.hrodqu 2024-04-06T03:30:18.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:18 smithi195 ceph-mon[53157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:30:18.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:18 smithi195 ceph-mon[53157]: osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474] boot 2024-04-06T03:30:18.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:18 smithi195 ceph-mon[53157]: osdmap e153: 8 total, 8 up, 8 in 2024-04-06T03:30:18.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:18 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:30:18.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:18 smithi060 ceph-mon[67116]: Upgrade: It appears safe to stop mds.foofs.smithi060.hrodqu 2024-04-06T03:30:18.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:18 smithi060 ceph-mon[67116]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T03:30:18.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:18 smithi060 ceph-mon[67116]: osd.7 [v2:172.21.15.195:6824/973250474,v1:172.21.15.195:6825/973250474] boot 2024-04-06T03:30:18.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:18 smithi060 ceph-mon[67116]: osdmap e153: 8 total, 8 up, 8 in 2024-04-06T03:30:18.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:18 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:30:19.637 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:19 smithi060 ceph-mon[67116]: pgmap v307: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 567 B/s rd, 1.5 KiB/s wr, 1 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:19.638 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:19.638 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:30:19.638 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:19.638 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:19 smithi060 ceph-mon[67116]: osdmap e154: 8 total, 8 up, 8 in 2024-04-06T03:30:19.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:19 smithi195 ceph-mon[53157]: pgmap v307: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 223 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 567 B/s rd, 1.5 KiB/s wr, 1 op/s; 69/426 objects degraded (16.197%) 2024-04-06T03:30:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi060.hrodqu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:30:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:19 smithi195 ceph-mon[53157]: osdmap e154: 8 total, 8 up, 8 in 2024-04-06T03:30:20.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:20 smithi060 ceph-mon[67116]: Upgrade: Updating mds.foofs.smithi060.hrodqu 2024-04-06T03:30:20.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:20 smithi060 ceph-mon[67116]: Deploying daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:30:20.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:20 smithi195 ceph-mon[53157]: Upgrade: Updating mds.foofs.smithi060.hrodqu 2024-04-06T03:30:20.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:20 smithi195 ceph-mon[53157]: Deploying daemon mds.foofs.smithi060.hrodqu on smithi060 2024-04-06T03:30:21.138 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[67112]: 2024-04-06T03:30:21.113+0000 7f8c5252a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:21.389 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:30:21.137+0000 7fbcabd08700 -1 bdev(0x559cd59fc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 11 2024-04-06T03:30:21.389 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:30:21.137+0000 7fbca64fd700 -1 bdev(0x559cd59fc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 14 2024-04-06T03:30:21.389 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:30:21.138+0000 7fbcaa505700 -1 bdev(0x559cd59fc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 14 2024-04-06T03:30:21.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: pgmap v310: 97 pgs: 6 active+undersized, 27 active+undersized+degraded, 64 active+clean; 223 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 57/426 objects degraded (13.380%); 0 B/s, 5 objects/s recovering 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Health check update: Degraded data redundancy: 57/426 objects degraded (13.380%), 27 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: osdmap e155: 8 total, 8 up, 8 in 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Standby daemon mds.foofs.smithi195.dtvkez assigned to filesystem foofs as rank 0 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-06T03:30:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:21 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:replay} 2024-04-06T03:30:21.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: pgmap v310: 97 pgs: 6 active+undersized, 27 active+undersized+degraded, 64 active+clean; 223 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 57/426 objects degraded (13.380%); 0 B/s, 5 objects/s recovering 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Health check update: Degraded data redundancy: 57/426 objects degraded (13.380%), 27 pgs degraded (PG_DEGRADED) 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: osdmap e155: 8 total, 8 up, 8 in 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Standby daemon mds.foofs.smithi195.dtvkez assigned to filesystem foofs as rank 0 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-06T03:30:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:21 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:replay} 2024-04-06T03:30:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:23 smithi195 ceph-mon[53157]: pgmap v312: 97 pgs: 2 active+undersized, 13 active+undersized+degraded, 82 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 5 op/s; 22/426 objects degraded (5.164%); 281 KiB/s, 9 objects/s recovering 2024-04-06T03:30:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:23 smithi195 ceph-mon[53157]: reconnect by client.24331 172.21.15.195:0/3527756136 after 0 2024-04-06T03:30:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:23 smithi195 ceph-mon[53157]: reconnect by client.14542 172.21.15.60:0/1152386077 after 0.000999982 2024-04-06T03:30:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:23 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:reconnect 2024-04-06T03:30:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:23 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:reconnect} 2024-04-06T03:30:23.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:23 smithi060 ceph-mon[67116]: pgmap v312: 97 pgs: 2 active+undersized, 13 active+undersized+degraded, 82 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 5 op/s; 22/426 objects degraded (5.164%); 281 KiB/s, 9 objects/s recovering 2024-04-06T03:30:23.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:23 smithi060 ceph-mon[67116]: reconnect by client.24331 172.21.15.195:0/3527756136 after 0 2024-04-06T03:30:23.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:23 smithi060 ceph-mon[67116]: reconnect by client.14542 172.21.15.60:0/1152386077 after 0.000999982 2024-04-06T03:30:23.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:23 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:reconnect 2024-04-06T03:30:23.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:23 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:reconnect} 2024-04-06T03:30:24.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:24 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:rejoin 2024-04-06T03:30:24.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:24 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:rejoin} 2024-04-06T03:30:24.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:24 smithi195 ceph-mon[53157]: daemon mds.foofs.smithi195.dtvkez is now active in filesystem foofs as rank 0 2024-04-06T03:30:24.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:24 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:rejoin 2024-04-06T03:30:24.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:24 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi195.dtvkez=up:rejoin} 2024-04-06T03:30:24.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:24 smithi060 ceph-mon[67116]: daemon mds.foofs.smithi195.dtvkez is now active in filesystem foofs as rank 0 2024-04-06T03:30:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:25 smithi195 ceph-mon[53157]: pgmap v313: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 6 op/s; 233 KiB/s, 20 objects/s recovering 2024-04-06T03:30:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:25 smithi195 ceph-mon[53157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/426 objects degraded (5.164%), 13 pgs degraded) 2024-04-06T03:30:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:25 smithi195 ceph-mon[53157]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T03:30:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:25 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:active 2024-04-06T03:30:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:25 smithi195 ceph-mon[53157]: fsmap foofs:1 {0=foofs.smithi195.dtvkez=up:active} 2024-04-06T03:30:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:25 smithi060 ceph-mon[67116]: pgmap v313: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 6 op/s; 233 KiB/s, 20 objects/s recovering 2024-04-06T03:30:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:25 smithi060 ceph-mon[67116]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/426 objects degraded (5.164%), 13 pgs degraded) 2024-04-06T03:30:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:25 smithi060 ceph-mon[67116]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T03:30:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:25 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.195:6832/3389285593,v1:172.21.15.195:6833/3389285593] up:active 2024-04-06T03:30:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:25 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi195.dtvkez=up:active} 2024-04-06T03:30:27.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:27 smithi195 ceph-mon[53157]: pgmap v314: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8 op/s; 176 KiB/s, 15 objects/s recovering 2024-04-06T03:30:27.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:27 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:27.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:27 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:27.748 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:27 smithi060 ceph-mon[67116]: pgmap v314: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8 op/s; 176 KiB/s, 15 objects/s recovering 2024-04-06T03:30:27.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:27 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:27.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:27 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:29.702 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:29 smithi060 ceph-mon[67116]: pgmap v315: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 228 B/s wr, 12 op/s; 156 KiB/s, 13 objects/s recovering 2024-04-06T03:30:29.722 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:29 smithi195 ceph-mon[53157]: pgmap v315: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 228 B/s wr, 12 op/s; 156 KiB/s, 13 objects/s recovering 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: pgmap v316: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s; 140 KiB/s, 10 objects/s recovering 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:boot 2024-04-06T03:30:31.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi195.dtvkez=up:active} 1 up:standby 2024-04-06T03:30:31.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:31 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: pgmap v316: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s; 140 KiB/s, 10 objects/s recovering 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:boot 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: fsmap foofs:1 {0=foofs.smithi195.dtvkez=up:active} 1 up:standby 2024-04-06T03:30:31.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:31 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:30:33.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:33 smithi195 ceph-mon[53157]: pgmap v317: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.8 KiB/s wr, 13 op/s; 127 KiB/s, 8 objects/s recovering 2024-04-06T03:30:33.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:33.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:33 smithi060 ceph-mon[67116]: pgmap v317: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.8 KiB/s wr, 13 op/s; 127 KiB/s, 8 objects/s recovering 2024-04-06T03:30:33.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:35.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:35 smithi195 ceph-mon[53157]: pgmap v318: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 10 op/s; 0 B/s, 6 objects/s recovering 2024-04-06T03:30:35.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:35.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:35.498 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:35 smithi060 ceph-mon[67116]: pgmap v318: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 10 op/s; 0 B/s, 6 objects/s recovering 2024-04-06T03:30:35.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:35.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:35.558 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (9m) 1s ago 19m 25.4M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (6m) 1s ago 19m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (5m) 24s ago 18m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 1s ago 18m - - 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (6s) 1s ago 13m 12.4M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (13m) 24s ago 13m 16.5M - 17.2.0 e1d6a67b021e cec019ce64f1 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (8m) 1s ago 20m 511M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (7m) 24s ago 18m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (7m) 1s ago 20m 62.8M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (7m) 24s ago 18m 48.0M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:30:36.266 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (12m) 1s ago 12m 277M - 4.0 e1d6a67b021e edd17339e7f5 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (12m) 24s ago 12m 83.7M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (9m) 1s ago 18m 20.6M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (9m) 24s ago 18m 19.6M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (5m) 1s ago 17m 399M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (4m) 1s ago 17m 377M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (3m) 1s ago 16m 359M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (3m) 1s ago 16m 416M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (2m) 24s ago 15m 340M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (104s) 24s ago 15m 314M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (67s) 24s ago 15m 289M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (29s) 24s ago 14m 12.6M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:30:36.267 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (8m) 1s ago 17m 67.8M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:30:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 13 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:30:36.710 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "osd", 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "crash" 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:30:37.076 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "15/23 daemons upgraded", 2024-04-06T03:30:37.077 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading mds daemons", 2024-04-06T03:30:37.077 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:30:37.077 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: pgmap v319: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:37 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1364605618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: pgmap v319: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:37.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:37 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1364605618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:37.489 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:30:37.489 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:30:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:30:38.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:38 smithi195 ceph-mon[53157]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:38.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:38 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3332539243' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:30:38.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:38 smithi060 ceph-mon[67116]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:30:38.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:38 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3332539243' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:30:39.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:39 smithi060 ceph-mon[67116]: pgmap v320: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-06T03:30:39.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:39 smithi195 ceph-mon[53157]: pgmap v320: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-06T03:30:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: Detected new or changed devices on smithi060 2024-04-06T03:30:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: pgmap v321: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: pgmap v322: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 8.6 KiB/s wr, 3 op/s 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi195.dtvkez"]}]: dispatch 2024-04-06T03:30:41.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:40 smithi195 ceph-mon[53157]: Upgrade: It appears safe to stop mds.foofs.smithi195.dtvkez 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: Detected new or changed devices on smithi060 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: pgmap v321: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: pgmap v322: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 8.6 KiB/s wr, 3 op/s 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:41.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi195.dtvkez"]}]: dispatch 2024-04-06T03:30:41.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:40 smithi060 ceph-mon[67116]: Upgrade: It appears safe to stop mds.foofs.smithi195.dtvkez 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: Upgrade: Updating mds.foofs.smithi195.dtvkez 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: Deploying daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:30:42.910 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: pgmap v323: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 210 B/s wr, 0 op/s 2024-04-06T03:30:42.911 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:42.911 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:42.998 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: Upgrade: Updating mds.foofs.smithi195.dtvkez 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi195.dtvkez", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: Deploying daemon mds.foofs.smithi195.dtvkez on smithi195 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: pgmap v323: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 210 B/s wr, 0 op/s 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:42.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:43.999 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:30:43.806+0000 7fbcaa505700 -1 bdev(0x559cd59fc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 10 2024-04-06T03:30:43.999 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:30:43.806+0000 7fbca64fd700 -1 bdev(0x559cd59fc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 9 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[67112]: 2024-04-06T03:30:43.731+0000 7f8c5252a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: osdmap e156: 8 total, 8 up, 8 in 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: Standby daemon mds.foofs.smithi060.hrodqu assigned to filesystem foofs as rank 0 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T03:30:43.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:30:44.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-06T03:30:44.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:43 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:replay} 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: osdmap e156: 8 total, 8 up, 8 in 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: Standby daemon mds.foofs.smithi060.hrodqu assigned to filesystem foofs as rank 0 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-06T03:30:44.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:43 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:replay} 2024-04-06T03:30:45.222 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:44 smithi195 ceph-mon[53157]: pgmap v325: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 265 B/s rd, 397 B/s wr, 0 op/s 2024-04-06T03:30:45.248 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:44 smithi060 ceph-mon[67116]: pgmap v325: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 265 B/s rd, 397 B/s wr, 0 op/s 2024-04-06T03:30:47.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:46 smithi195 ceph-mon[53157]: reconnect by client.14542 172.21.15.60:0/1152386077 after 0.00099999 2024-04-06T03:30:47.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:46 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:reconnect 2024-04-06T03:30:47.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:46 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:reconnect} 2024-04-06T03:30:47.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:46 smithi195 ceph-mon[53157]: reconnect by client.24331 172.21.15.195:0/3527756136 after 0.0629993 2024-04-06T03:30:47.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:46 smithi195 ceph-mon[53157]: pgmap v326: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 397 B/s wr, 3 op/s 2024-04-06T03:30:47.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:46 smithi060 ceph-mon[67116]: reconnect by client.14542 172.21.15.60:0/1152386077 after 0.00099999 2024-04-06T03:30:47.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:46 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:reconnect 2024-04-06T03:30:47.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:46 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:reconnect} 2024-04-06T03:30:47.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:46 smithi060 ceph-mon[67116]: reconnect by client.24331 172.21.15.195:0/3527756136 after 0.0629993 2024-04-06T03:30:47.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:46 smithi060 ceph-mon[67116]: pgmap v326: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 397 B/s wr, 3 op/s 2024-04-06T03:30:48.164 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:47 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:rejoin 2024-04-06T03:30:48.164 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:47 smithi195 ceph-mon[53157]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:rejoin} 2024-04-06T03:30:48.165 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:47 smithi195 ceph-mon[53157]: daemon mds.foofs.smithi060.hrodqu is now active in filesystem foofs as rank 0 2024-04-06T03:30:48.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:47 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:rejoin 2024-04-06T03:30:48.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:47 smithi060 ceph-mon[67116]: fsmap foofs:1/1 {0=foofs.smithi060.hrodqu=up:rejoin} 2024-04-06T03:30:48.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:47 smithi060 ceph-mon[67116]: daemon mds.foofs.smithi060.hrodqu is now active in filesystem foofs as rank 0 2024-04-06T03:30:49.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:48 smithi195 ceph-mon[53157]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T03:30:49.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:48 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:active 2024-04-06T03:30:49.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:48 smithi195 ceph-mon[53157]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 2024-04-06T03:30:49.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:48 smithi195 ceph-mon[53157]: pgmap v327: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 511 B/s wr, 8 op/s 2024-04-06T03:30:49.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:48 smithi060 ceph-mon[67116]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T03:30:49.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:48 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819] up:active 2024-04-06T03:30:49.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:48 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 2024-04-06T03:30:49.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:48 smithi060 ceph-mon[67116]: pgmap v327: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 511 B/s wr, 8 op/s 2024-04-06T03:30:51.194 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:50 smithi195 ceph-mon[53157]: pgmap v328: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 8 op/s 2024-04-06T03:30:51.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:50 smithi060 ceph-mon[67116]: pgmap v328: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 8 op/s 2024-04-06T03:30:52.347 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: mds.? [v2:172.21.15.195:6832/3045290688,v1:172.21.15.195:6833/3045290688] up:boot 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:30:52.348 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:52 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: mds.? [v2:172.21.15.195:6832/3045290688,v1:172.21.15.195:6833/3045290688] up:boot 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: fsmap foofs:1 {0=foofs.smithi060.hrodqu=up:active} 1 up:standby 2024-04-06T03:30:52.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:52 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:30:53.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:53 smithi195 ceph-mon[53157]: pgmap v329: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 8.8 KiB/s wr, 13 op/s 2024-04-06T03:30:53.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:53 smithi060 ceph-mon[67116]: pgmap v329: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 8.8 KiB/s wr, 13 op/s 2024-04-06T03:30:55.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:55 smithi195 ceph-mon[53157]: pgmap v330: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.6 KiB/s wr, 14 op/s 2024-04-06T03:30:55.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:55 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:55.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:55 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:55.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:55 smithi060 ceph-mon[67116]: pgmap v330: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.6 KiB/s wr, 14 op/s 2024-04-06T03:30:55.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:55 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:55.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:55 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:56.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:56.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:56 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:56.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:56.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:56 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:57 smithi195 ceph-mon[53157]: pgmap v331: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-04-06T03:30:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:57.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:57 smithi060 ceph-mon[67116]: pgmap v331: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-04-06T03:30:57.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:59.702 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: pgmap v332: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi060.hrodqu"}]': finished 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi195.dtvkez"}]': finished 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.704 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.705 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.705 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.705 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.705 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:30:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: pgmap v332: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi060.hrodqu"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi060.hrodqu"}]': finished 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi195.dtvkez"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi195.dtvkez"}]': finished 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:30:59.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:30:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Detected new or changed devices on smithi195 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: pgmap v333: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: pgmap v334: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 9.6 KiB/s wr, 7 op/s 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all mds 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all rgw 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T03:31:00.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:00 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all iscsi 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Detected new or changed devices on smithi195 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: pgmap v333: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: pgmap v334: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 9.6 KiB/s wr, 7 op/s 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all mds 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all rgw 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T03:31:00.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:00 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all iscsi 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: pgmap v335: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Upgrade: Updating nfs.foo.0.0.smithi060.natxue (1/2) 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Creating key for client.nfs.foo.0.0.smithi060.natxue 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-06T03:31:02.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Rados config object exists: conf-nfs.foo 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Creating key for client.nfs.foo.0.0.smithi060.natxue-rgw 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Bind address in nfs.foo.0.0.smithi060.natxue's ganesha conf is defaulting to empty 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:02.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:01 smithi060 ceph-mon[67116]: Deploying daemon nfs.foo.0.0.smithi060.natxue on smithi060 2024-04-06T03:31:02.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: pgmap v335: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:31:02.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Upgrade: Updating nfs.foo.0.0.smithi060.natxue (1/2) 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Creating key for client.nfs.foo.0.0.smithi060.natxue 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Rados config object exists: conf-nfs.foo 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Creating key for client.nfs.foo.0.0.smithi060.natxue-rgw 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi060.natxue-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Bind address in nfs.foo.0.0.smithi060.natxue's ganesha conf is defaulting to empty 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:02.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:01 smithi195 ceph-mon[53157]: Deploying daemon nfs.foo.0.0.smithi060.natxue on smithi060 2024-04-06T03:31:04.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:03 smithi060 ceph-mon[67116]: pgmap v336: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 456 B/s rd, 228 B/s wr, 0 op/s 2024-04-06T03:31:04.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:03 smithi195 ceph-mon[53157]: pgmap v336: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 456 B/s rd, 228 B/s wr, 0 op/s 2024-04-06T03:31:06.044 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:05 smithi195 ceph-mon[53157]: pgmap v337: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 456 B/s rd, 228 B/s wr, 0 op/s 2024-04-06T03:31:06.044 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:05 smithi195 ceph-mon[53157]: Evicting (and blocklisting) client session 14542 (172.21.15.60:0/1152386077) 2024-04-06T03:31:06.044 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:05 smithi195 ceph-mon[53157]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.60:0/1152386077"}]: dispatch 2024-04-06T03:31:06.044 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:06.044 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:05 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:06.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:05 smithi060 ceph-mon[67116]: pgmap v337: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 456 B/s rd, 228 B/s wr, 0 op/s 2024-04-06T03:31:06.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:05 smithi060 ceph-mon[67116]: Evicting (and blocklisting) client session 14542 (172.21.15.60:0/1152386077) 2024-04-06T03:31:06.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:05 smithi060 ceph-mon[67116]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.60:0/1152386077"}]: dispatch 2024-04-06T03:31:06.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:06.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:05 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.60:0/1152386077"}]': finished 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: osdmap e157: 8 total, 8 up, 8 in 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:31:07.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:07.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:31:07.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:31:07.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:31:07.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:06 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.60:0/1152386077"}]': finished 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: osdmap e157: 8 total, 8 up, 8 in 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:07.457 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-06T03:31:07.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-06T03:31:07.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi195.rdvmyy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:31:07.458 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:06 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Upgrade: Updating nfs.foo.1.0.smithi195.rdvmyy (2/2) 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: pgmap v339: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Rados config object exists: conf-nfs.foo 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy-rgw 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Bind address in nfs.foo.1.0.smithi195.rdvmyy's ganesha conf is defaulting to empty 2024-04-06T03:31:08.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:07 smithi195 ceph-mon[53157]: Deploying daemon nfs.foo.1.0.smithi195.rdvmyy on smithi195 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Upgrade: Updating nfs.foo.1.0.smithi195.rdvmyy (2/2) 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: pgmap v339: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Rados config object exists: conf-nfs.foo 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Creating key for client.nfs.foo.1.0.smithi195.rdvmyy-rgw 2024-04-06T03:31:08.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Bind address in nfs.foo.1.0.smithi195.rdvmyy's ganesha conf is defaulting to empty 2024-04-06T03:31:08.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:07 smithi060 ceph-mon[67116]: Deploying daemon nfs.foo.1.0.smithi195.rdvmyy on smithi195 2024-04-06T03:31:08.643 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (10m) 33s ago 19m 25.4M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (6m) 33s ago 19m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (6m) 14s ago 18m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 33s ago 19m - - 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (39s) 33s ago 13m 12.4M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (17s) 14s ago 13m 11.2M - 19.0.0-2027-g9f192350 a5511c7deba9 84b07e74e933 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (9m) 33s ago 20m 511M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (8m) 14s ago 18m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (8m) 33s ago 21m 62.8M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (7m) 14s ago 18m 48.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 starting - - - - 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (13m) 14s ago 13m 83.8M - 4.0 e1d6a67b021e 687c8910d986 2024-04-06T03:31:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (10m) 33s ago 19m 20.6M - 1.3.1 1dbe0e931976 d3c4dc58ce3a 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (9m) 14s ago 18m 17.6M - 1.3.1 1dbe0e931976 5c9e21ec1fbe 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (5m) 33s ago 18m 399M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (5m) 33s ago 17m 377M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (4m) 33s ago 17m 359M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (3m) 33s ago 16m 416M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (2m) 14s ago 16m 342M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (2m) 14s ago 16m 317M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (100s) 14s ago 15m 291M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (61s) 14s ago 15m 278M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:31:08.645 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (8m) 33s ago 18m 67.8M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:31:09.055 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:31:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 14 2024-04-06T03:31:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:31:09.057 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:31:09.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:09 smithi195 ceph-mon[53157]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:09.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:09 smithi195 ceph-mon[53157]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "mds", 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:31:09.413 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout: "osd" 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "16/23 daemons upgraded", 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading nfs daemons", 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:31:09.414 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:31:09.414 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:09 smithi060 ceph-mon[67116]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:09.414 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:09 smithi060 ceph-mon[67116]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:09.827 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:31:09.828 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:31:09.828 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:31:10.335 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:10 smithi195 ceph-mon[53157]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:10.335 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:10 smithi195 ceph-mon[53157]: pgmap v340: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:10.335 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:10 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/1640845027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:10.335 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:10 smithi195 ceph-mon[53157]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:10.335 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:10 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/2022360219' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:31:10.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:10 smithi060 ceph-mon[67116]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:10.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:10 smithi060 ceph-mon[67116]: pgmap v340: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:10.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:10 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/1640845027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:10.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:10 smithi060 ceph-mon[67116]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:10.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:10 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2022360219' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:31:11.705 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:11.706 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:11.706 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:11 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:11.706 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:11 smithi195 ceph-mon[53157]: pgmap v341: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:11.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:11.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:11.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:11 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:11.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:11 smithi060 ceph-mon[67116]: pgmap v341: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-06T03:31:12.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:12.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:13.648 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:13 smithi195 ceph-mon[53157]: pgmap v342: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-06T03:31:13.648 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:13 smithi195 ceph-mon[53157]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.195:0/3527756136"}]: dispatch 2024-04-06T03:31:13.929 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:13 smithi060 ceph-mon[67116]: pgmap v342: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-06T03:31:13.929 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:13 smithi060 ceph-mon[67116]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.195:0/3527756136"}]: dispatch 2024-04-06T03:31:14.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:14 smithi195 ceph-mon[53157]: Evicting (and blocklisting) client session 24331 (172.21.15.195:0/3527756136) 2024-04-06T03:31:14.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:14 smithi195 ceph-mon[53157]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.195:0/3527756136"}]': finished 2024-04-06T03:31:14.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:14 smithi195 ceph-mon[53157]: osdmap e158: 8 total, 8 up, 8 in 2024-04-06T03:31:14.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:14 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:14.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:14 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:14.982 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:14 smithi060 ceph-mon[67116]: Evicting (and blocklisting) client session 24331 (172.21.15.195:0/3527756136) 2024-04-06T03:31:14.982 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:14 smithi060 ceph-mon[67116]: from='mds.? [v2:172.21.15.60:6834/2788659819,v1:172.21.15.60:6835/2788659819]' entity='mds.foofs.smithi060.hrodqu' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.195:0/3527756136"}]': finished 2024-04-06T03:31:14.982 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:14 smithi060 ceph-mon[67116]: osdmap e158: 8 total, 8 up, 8 in 2024-04-06T03:31:14.983 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:14 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:14.983 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:14 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:15 smithi195 ceph-mon[53157]: pgmap v344: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-04-06T03:31:15.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:15 smithi060 ceph-mon[67116]: pgmap v344: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-04-06T03:31:15.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:15.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:17.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:17.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:17 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:17.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:17 smithi195 ceph-mon[53157]: pgmap v345: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-06T03:31:17.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:17.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:17 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:17.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:17 smithi060 ceph-mon[67116]: pgmap v345: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-06T03:31:19.179 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:19.179 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:19 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:19.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:19.468 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:19 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:20.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:20 smithi060 ceph-mon[67116]: pgmap v346: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-06T03:31:20.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:20 smithi195 ceph-mon[53157]: pgmap v346: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: pgmap v347: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: pgmap v348: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 131 B/s rd, 527 B/s wr, 1 op/s 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:21.654 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all nfs 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi060.natxue"}]: dispatch 2024-04-06T03:31:21.655 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi060.natxue"}]': finished 2024-04-06T03:31:21.656 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi195.rdvmyy"}]: dispatch 2024-04-06T03:31:21.656 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi195.rdvmyy"}]': finished 2024-04-06T03:31:21.656 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.656 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: Upgrade: Setting container_image for all nvmeof 2024-04-06T03:31:21.656 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:21 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:21.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: pgmap v347: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: pgmap v348: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 131 B/s rd, 527 B/s wr, 1 op/s 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all nfs 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi060.natxue"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi060.natxue"}]': finished 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi195.rdvmyy"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi195.rdvmyy"}]': finished 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: Upgrade: Setting container_image for all nvmeof 2024-04-06T03:31:21.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:21 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:22.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:22 smithi060 ceph-mon[67116]: Upgrade: Updating node-exporter.smithi060 (1/2) 2024-04-06T03:31:22.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:22 smithi060 ceph-mon[67116]: Deploying daemon node-exporter.smithi060 on smithi060 2024-04-06T03:31:22.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:22 smithi195 ceph-mon[53157]: Upgrade: Updating node-exporter.smithi060 (1/2) 2024-04-06T03:31:22.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:22 smithi195 ceph-mon[53157]: Deploying daemon node-exporter.smithi060 on smithi060 2024-04-06T03:31:23.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:23 smithi195 ceph-mon[53157]: pgmap v349: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 113 B/s rd, 455 B/s wr, 0 op/s 2024-04-06T03:31:23.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:23 smithi060 ceph-mon[67116]: pgmap v349: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 113 B/s rd, 455 B/s wr, 0 op/s 2024-04-06T03:31:25.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:25 smithi195 ceph-mon[53157]: pgmap v350: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 419 B/s wr, 0 op/s 2024-04-06T03:31:25.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:25 smithi060 ceph-mon[67116]: pgmap v350: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 419 B/s wr, 0 op/s 2024-04-06T03:31:27.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:27 smithi195 ceph-mon[53157]: pgmap v351: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 209 B/s rd, 0 B/s wr, 0 op/s 2024-04-06T03:31:27.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:27 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:27.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:27 smithi060 ceph-mon[67116]: pgmap v351: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 209 B/s rd, 0 B/s wr, 0 op/s 2024-04-06T03:31:27.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:27 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:29.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:29 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:29.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:29 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:29.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:29 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:29.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:29 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:30.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:30 smithi195 ceph-mon[53157]: pgmap v352: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 209 B/s rd, 0 op/s 2024-04-06T03:31:30.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:30 smithi060 ceph-mon[67116]: pgmap v352: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 209 B/s rd, 0 op/s 2024-04-06T03:31:31.389 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:31 smithi195 ceph-mon[53157]: Upgrade: Updating node-exporter.smithi195 (2/2) 2024-04-06T03:31:31.390 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:31 smithi195 ceph-mon[53157]: Deploying daemon node-exporter.smithi195 on smithi195 2024-04-06T03:31:31.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:31 smithi060 ceph-mon[67116]: Upgrade: Updating node-exporter.smithi195 (2/2) 2024-04-06T03:31:31.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:31 smithi060 ceph-mon[67116]: Deploying daemon node-exporter.smithi195 on smithi195 2024-04-06T03:31:32.441 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:32 smithi195 ceph-mon[53157]: pgmap v353: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:31:32.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:32 smithi060 ceph-mon[67116]: pgmap v353: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:31:34.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:34 smithi195 ceph-mon[53157]: pgmap v354: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:34.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:34 smithi060 ceph-mon[67116]: pgmap v354: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:35 smithi195 ceph-mon[53157]: pgmap v355: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:35.966 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:35 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:35.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:35 smithi060 ceph-mon[67116]: pgmap v355: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:35.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:35.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:35.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:35 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:38.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:37 smithi195 ceph-mon[53157]: pgmap v356: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:31:38.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:37 smithi060 ceph-mon[67116]: pgmap v356: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:31:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:40.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:39 smithi195 ceph-mon[53157]: pgmap v357: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:40.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:40.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:39 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:40.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:39 smithi060 ceph-mon[67116]: pgmap v357: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:40.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:40.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:39 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:40.266 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (11m) 1s ago 20m 25.4M - 0.23.0 ba2b418f427c c6eb29b01a2f 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (7m) 1s ago 20m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (7m) 3s ago 19m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 1s ago 19m - - 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (71s) 1s ago 14m 33.5M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (50s) 3s ago 14m 12.6M - 19.0.0-2027-g9f192350 a5511c7deba9 84b07e74e933 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (9m) 1s ago 21m 514M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (9m) 3s ago 19m 453M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (8m) 1s ago 21m 63.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:31:41.083 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (8m) 3s ago 19m 50.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (36s) 1s ago 13m 72.5M - 5.7 a5511c7deba9 ba6fd13b0046 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (30s) 3s ago 13m 63.8M - 5.7 a5511c7deba9 9a320e957771 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (13s) 1s ago 19m 3519k - 1.5.0 0da6a335fe13 fcbd9c3db25f 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (6s) 3s ago 19m 3263k - 1.5.0 0da6a335fe13 4c932fd5d566 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (6m) 1s ago 18m 399M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (5m) 1s ago 18m 378M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (4m) 1s ago 17m 360M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (4m) 1s ago 17m 420M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (3m) 3s ago 16m 343M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (2m) 3s ago 16m 316M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (2m) 3s ago 16m 291M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (93s) 3s ago 15m 279M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:31:41.084 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (9m) 1s ago 19m 69.9M - 2.33.4 514e6a882f6e 897c95880553 2024-04-06T03:31:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:40 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:40 smithi195 ceph-mon[53157]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:40 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:40 smithi060 ceph-mon[67116]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 14 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:31:41.529 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:31:41.890 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:31:41.890 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "mds", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "osd", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "nfs" 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "18/23 daemons upgraded", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:31:41.891 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:31:42.349 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:31:42.349 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:31:42.349 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:31:42.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: pgmap v358: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:42.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:42.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:42.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:42.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:42.474 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:42 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3060251212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: pgmap v358: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:42.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:42 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3060251212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:43.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:43 smithi060 ceph-mon[67116]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:43.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:43.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:43.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:43 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/935187601' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:31:43.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:43 smithi195 ceph-mon[53157]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:31:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:43 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/935187601' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:31:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:44.388 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:44 smithi060 ceph-mon[67116]: pgmap v359: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:44.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:44 smithi195 ceph-mon[53157]: pgmap v359: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: pgmap v360: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: pgmap v361: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: pgmap v362: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.751 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:45 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: pgmap v360: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:45.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: pgmap v361: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: pgmap v362: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:45.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:45 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:31:47.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:47 smithi060 ceph-mon[67116]: Upgrade: Updating prometheus.smithi060 2024-04-06T03:31:47.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:47 smithi060 ceph-mon[67116]: pgmap v363: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:47.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:47 smithi060 ceph-mon[67116]: Deploying daemon prometheus.smithi060 on smithi060 2024-04-06T03:31:48.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:47 smithi195 ceph-mon[53157]: Upgrade: Updating prometheus.smithi060 2024-04-06T03:31:48.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:47 smithi195 ceph-mon[53157]: pgmap v363: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:48.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:47 smithi195 ceph-mon[53157]: Deploying daemon prometheus.smithi060 on smithi060 2024-04-06T03:31:49.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:49 smithi060 ceph-mon[67116]: pgmap v364: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:50.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:49 smithi195 ceph-mon[53157]: pgmap v364: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:52.050 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:51 smithi060 ceph-mon[67116]: pgmap v365: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:52.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:51 smithi195 ceph-mon[53157]: pgmap v365: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:53.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:53.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: pgmap v366: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:53.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:53 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: pgmap v366: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:53.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:53 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:56.199 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:55 smithi060 ceph-mon[67116]: pgmap v367: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:31:56.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:55 smithi195 ceph-mon[53157]: pgmap v367: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:31:58.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:57 smithi195 ceph-mon[53157]: pgmap v368: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:31:58.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:58.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:58.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:58.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:57 smithi060 ceph-mon[67116]: pgmap v368: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:31:58.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:58.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:58.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:59.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:59 smithi060 ceph-mon[67116]: pgmap v369: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:31:59.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:31:59.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:31:59 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:00.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:59 smithi195 ceph-mon[53157]: pgmap v369: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:00.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:00.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:31:59 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:01.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:01 smithi195 ceph-mon[53157]: pgmap v370: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:01.988 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:01 smithi060 ceph-mon[67116]: pgmap v370: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:03.350 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: pgmap v371: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: pgmap v372: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.351 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.352 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:03 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: pgmap v371: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: pgmap v372: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:03.725 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:03 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:04.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:04 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:32:04.732 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:04 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:32:05.604 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:05 smithi060 ceph-mon[67116]: Upgrade: Updating alertmanager.smithi060 2024-04-06T03:32:05.605 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:05 smithi060 ceph-mon[67116]: Deploying daemon alertmanager.smithi060 on smithi060 2024-04-06T03:32:05.605 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:05 smithi060 ceph-mon[67116]: pgmap v373: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:05.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:05 smithi195 ceph-mon[53157]: Upgrade: Updating alertmanager.smithi060 2024-04-06T03:32:05.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:05 smithi195 ceph-mon[53157]: Deploying daemon alertmanager.smithi060 on smithi060 2024-04-06T03:32:05.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:05 smithi195 ceph-mon[53157]: pgmap v373: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:07.662 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:07 smithi060 ceph-mon[67116]: pgmap v374: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:07.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:07 smithi195 ceph-mon[53157]: pgmap v374: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:09.624 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:09 smithi060 ceph-mon[67116]: pgmap v375: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:09.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:09 smithi195 ceph-mon[53157]: pgmap v375: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:10.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:10.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:11.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: pgmap v376: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:11.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:10 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:11.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: pgmap v376: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 212 B/s rd, 0 op/s 2024-04-06T03:32:11.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:11.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:10 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:12.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:12.727 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:32:12.727 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 starting - - - - 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (7m) 15s ago 20m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (7m) 35s ago 20m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 error 15s ago 20m - - 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (103s) 15s ago 14m 33.7M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (82s) 35s ago 14m 12.6M - 19.0.0-2027-g9f192350 a5511c7deba9 84b07e74e933 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (10m) 15s ago 21m 516M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (9m) 35s ago 20m 453M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (9m) 15s ago 22m 62.8M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (8m) 35s ago 19m 50.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:32:13.430 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (68s) 15s ago 14m 74.6M - 5.7 a5511c7deba9 ba6fd13b0046 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (63s) 35s ago 14m 63.8M - 5.7 a5511c7deba9 9a320e957771 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (45s) 15s ago 20m 8418k - 1.5.0 0da6a335fe13 fcbd9c3db25f 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (38s) 35s ago 19m 3263k - 1.5.0 0da6a335fe13 4c932fd5d566 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (7m) 15s ago 19m 400M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (6m) 15s ago 18m 379M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (5m) 15s ago 18m 359M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (4m) 15s ago 17m 420M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (3m) 35s ago 17m 343M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (3m) 35s ago 17m 316M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (2m) 35s ago 16m 291M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (2m) 35s ago 16m 279M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:32:13.431 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (21s) 15s ago 19m 45.8M - 2.43.0 a07b618ecd1d 664ec26b7b3e 2024-04-06T03:32:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:13 smithi195 ceph-mon[53157]: pgmap v377: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:32:13.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:13 smithi060 ceph-mon[67116]: pgmap v377: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:32:13.874 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:32:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 14 2024-04-06T03:32:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:32:13.876 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": true, 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [ 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "mds", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "mon", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "osd", 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "nfs" 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:32:14.266 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": "18/23 daemons upgraded", 2024-04-06T03:32:14.267 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-06T03:32:14.267 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:32:14.267 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:32:14.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:14 smithi060 ceph-mon[67116]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:14 smithi060 ceph-mon[67116]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:14 smithi060 ceph-mon[67116]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:14 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3000055657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:14.673 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T03:32:14.673 INFO:teuthology.orchestra.run.smithi060.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T03:32:14.674 INFO:teuthology.orchestra.run.smithi060.stdout: daemon grafana.smithi060 on smithi060 is in error state 2024-04-06T03:32:14.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:14 smithi195 ceph-mon[53157]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:14 smithi195 ceph-mon[53157]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:14 smithi195 ceph-mon[53157]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:14.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:14 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3000055657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:15.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:15 smithi195 ceph-mon[53157]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:15.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:15 smithi195 ceph-mon[53157]: pgmap v378: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:15.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:15 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3459090265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:32:15.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:15.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:15 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:15 smithi060 ceph-mon[67116]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:15 smithi060 ceph-mon[67116]: pgmap v378: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:15 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3459090265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:32:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:15 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:17.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:17.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:16 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:17.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:16 smithi060 ceph-mon[67116]: pgmap v379: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:32:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:16 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:17.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:16 smithi195 ceph-mon[53157]: pgmap v379: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:32:19.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:19 smithi060 ceph-mon[67116]: pgmap v380: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:19 smithi195 ceph-mon[53157]: pgmap v380: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:20.744 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: pgmap v381: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 180 B/s rd, 0 op/s 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: pgmap v382: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.745 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.746 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:20 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: pgmap v381: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 180 B/s rd, 0 op/s 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: pgmap v382: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:20.975 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:20 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:22.851 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:22 smithi060 ceph-mon[67116]: pgmap v383: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:22.851 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:22 smithi060 ceph-mon[67116]: Upgrade: Updating grafana.smithi060 2024-04-06T03:32:22.851 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:22 smithi060 ceph-mon[67116]: Deploying daemon grafana.smithi060 on smithi060 2024-04-06T03:32:22.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:22 smithi195 ceph-mon[53157]: pgmap v383: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:22.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:22 smithi195 ceph-mon[53157]: Upgrade: Updating grafana.smithi060 2024-04-06T03:32:22.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:22 smithi195 ceph-mon[53157]: Deploying daemon grafana.smithi060 on smithi060 2024-04-06T03:32:24.966 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:24 smithi060 ceph-mon[67116]: pgmap v384: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:24.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:24 smithi195 ceph-mon[53157]: pgmap v384: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:26.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:26 smithi060 ceph-mon[67116]: pgmap v385: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:32:26.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:27.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:27.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:26 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:27.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:26 smithi195 ceph-mon[53157]: pgmap v385: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-06T03:32:27.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:26 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:27.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:26 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:27.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:26 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:27.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:27 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:28.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:27 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:28.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:28 smithi060 ceph-mon[67116]: pgmap v386: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:29.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:28 smithi195 ceph-mon[53157]: pgmap v386: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 219 B/s rd, 0 op/s 2024-04-06T03:32:30.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:30 smithi060 ceph-mon[67116]: pgmap v387: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:32:31.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:30 smithi195 ceph-mon[53157]: pgmap v387: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-06T03:32:32.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:32 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:32.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:32 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:32.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:32 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:32.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:32 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:33.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:33 smithi060 ceph-mon[67116]: pgmap v388: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:33.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:33.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:33 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:33.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:33 smithi195 ceph-mon[53157]: pgmap v388: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:33.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:33.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:33 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:35.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:35 smithi060 ceph-mon[67116]: pgmap v389: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:35 smithi195 ceph-mon[53157]: pgmap v389: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: pgmap v390: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: pgmap v391: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:37.224 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.225 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: Upgrade: Finalizing container_image settings 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T03:32:37.226 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.227 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.228 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: Upgrade: Complete! 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:32:37.229 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:36 smithi195 ceph-mon[53157]: Cluster is now healthy 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: pgmap v390: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: pgmap v391: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.251 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: Upgrade: Finalizing container_image settings 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.252 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-06T03:32:37.253 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.254 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: Upgrade: Complete! 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.255 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T03:32:37.256 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:36 smithi060 ceph-mon[67116]: Cluster is now healthy 2024-04-06T03:32:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:38 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:39.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:38 smithi195 ceph-mon[53157]: pgmap v392: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:38 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:39.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:38 smithi060 ceph-mon[67116]: pgmap v392: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:41.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:40 smithi195 ceph-mon[53157]: pgmap v393: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:41.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:40 smithi060 ceph-mon[67116]: pgmap v393: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:43 smithi195 ceph-mon[53157]: pgmap v394: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:43.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:43 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:43.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:43 smithi060 ceph-mon[67116]: pgmap v394: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:43.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:32:43.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:43 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:45.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:45 smithi195 ceph-mon[53157]: pgmap v395: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:45.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:45 smithi060 ceph-mon[67116]: pgmap v395: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:45.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch ps' 2024-04-06T03:32:46.430 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:46 smithi060 ceph-mon[67116]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:46.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:46 smithi195 ceph-mon[53157]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:47.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:47 smithi195 ceph-mon[53157]: pgmap v396: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:47.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:47 smithi060 ceph-mon[67116]: pgmap v396: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-04-06T03:32:47.841 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (38s) 16s ago 21m 15.0M - 0.25.0 c8568f914cd2 63aaf983a76b 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (8m) 16s ago 21m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (8m) 69s ago 20m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (22s) 16s ago 21m 48.7M - 9.4.12 2bacad6d85d8 c68ca5acf547 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (2m) 16s ago 15m 33.9M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (117s) 69s ago 15m 12.6M - 19.0.0-2027-g9f192350 a5511c7deba9 84b07e74e933 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (11m) 16s ago 22m 523M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (10m) 69s ago 20m 453M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (9m) 16s ago 22m 63.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (9m) 69s ago 20m 50.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (103s) 16s ago 15m 74.6M - 5.7 a5511c7deba9 ba6fd13b0046 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (97s) 69s ago 14m 63.8M - 5.7 a5511c7deba9 9a320e957771 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (80s) 16s ago 20m 8262k - 1.5.0 0da6a335fe13 fcbd9c3db25f 2024-04-06T03:32:47.842 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (73s) 69s ago 20m 3263k - 1.5.0 0da6a335fe13 4c932fd5d566 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (7m) 16s ago 19m 400M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (6m) 16s ago 19m 379M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (6m) 16s ago 18m 360M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (5m) 16s ago 18m 421M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (4m) 69s ago 18m 343M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (3m) 69s ago 17m 316M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (3m) 69s ago 17m 291M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (2m) 69s ago 16m 279M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:32:47.843 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (55s) 16s ago 20m 46.8M - 2.43.0 a07b618ecd1d 664ec26b7b3e 2024-04-06T03:32:48.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions' 2024-04-06T03:32:49.432 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:49 smithi060 ceph-mon[67116]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:49.432 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:49 smithi060 ceph-mon[67116]: pgmap v397: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:32:49.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:49 smithi195 ceph-mon[53157]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:32:49.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:49 smithi195 ceph-mon[53157]: pgmap v397: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 14 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:32:50.805 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:32:51.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-06T03:32:51.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:51 smithi195 ceph-mon[53157]: pgmap v398: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:51.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:51 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/2950429380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:51.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:51 smithi060 ceph-mon[67116]: pgmap v398: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:51.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:51 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/2950429380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:32:53.435 INFO:teuthology.orchestra.run.smithi060.stdout:wait for servicemap items w/ changing names to refresh 2024-04-06T03:32:53.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:53 smithi060 ceph-mon[67116]: pgmap v399: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:53.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:53 smithi195 ceph-mon[53157]: pgmap v399: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:54.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'sleep 60' 2024-04-06T03:32:55.471 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:55 smithi060 ceph-mon[67116]: pgmap v400: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:55.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:55 smithi195 ceph-mon[53157]: pgmap v400: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:57 smithi195 ceph-mon[53157]: pgmap v401: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:57.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:57 smithi060 ceph-mon[67116]: pgmap v401: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:32:57.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:59.703 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:32:59 smithi060 ceph-mon[67116]: pgmap v402: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:32:59.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:32:59 smithi195 ceph-mon[53157]: pgmap v402: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:01.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:01 smithi195 ceph-mon[53157]: pgmap v403: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:01.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:01 smithi060 ceph-mon[67116]: pgmap v403: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:03.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:03 smithi195 ceph-mon[53157]: pgmap v404: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:03.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:03 smithi060 ceph-mon[67116]: pgmap v404: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:05.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:05 smithi195 ceph-mon[53157]: pgmap v405: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:05.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:05 smithi060 ceph-mon[67116]: pgmap v405: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:07.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:07 smithi195 ceph-mon[53157]: pgmap v406: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:07.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:07 smithi060 ceph-mon[67116]: pgmap v406: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:09.701 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:09 smithi060 ceph-mon[67116]: pgmap v407: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:09.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:09 smithi195 ceph-mon[53157]: pgmap v407: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:11.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:11 smithi195 ceph-mon[53157]: pgmap v408: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:11.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:11 smithi060 ceph-mon[67116]: pgmap v408: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:12.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:12.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:13.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:13 smithi195 ceph-mon[53157]: pgmap v409: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:13.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:13 smithi060 ceph-mon[67116]: pgmap v409: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:15.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:15 smithi195 ceph-mon[53157]: pgmap v410: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:15.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:15 smithi060 ceph-mon[67116]: pgmap v410: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:17.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:17 smithi195 ceph-mon[53157]: pgmap v411: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:17.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:17 smithi060 ceph-mon[67116]: pgmap v411: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:19.702 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:19 smithi060 ceph-mon[67116]: pgmap v412: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:19.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:19 smithi195 ceph-mon[53157]: pgmap v412: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:21.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:21 smithi195 ceph-mon[53157]: pgmap v413: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:21.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:21 smithi060 ceph-mon[67116]: pgmap v413: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:23.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:23 smithi195 ceph-mon[53157]: pgmap v414: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:23.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:23 smithi060 ceph-mon[67116]: pgmap v414: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:25.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:25 smithi195 ceph-mon[53157]: pgmap v415: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:25.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:25 smithi060 ceph-mon[67116]: pgmap v415: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:27.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:27 smithi195 ceph-mon[53157]: pgmap v416: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:27.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:27 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:27.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:27 smithi060 ceph-mon[67116]: pgmap v416: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:27.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:27 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:29.702 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:29 smithi060 ceph-mon[67116]: pgmap v417: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:29.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:29 smithi195 ceph-mon[53157]: pgmap v417: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:31.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:31 smithi195 ceph-mon[53157]: pgmap v418: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:31.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:31 smithi060 ceph-mon[67116]: pgmap v418: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:33.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:33 smithi195 ceph-mon[53157]: pgmap v419: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:33.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:33 smithi060 ceph-mon[67116]: pgmap v419: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:35.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:35 smithi195 ceph-mon[53157]: pgmap v420: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:35.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:35 smithi060 ceph-mon[67116]: pgmap v420: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:36.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:36 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:33:36.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:36 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:33:37.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:37 smithi195 ceph-mon[53157]: pgmap v421: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:37.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:33:37.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:33:37.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:33:37.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:37 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:33:37.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:37 smithi060 ceph-mon[67116]: pgmap v421: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:37.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:33:37.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:33:37.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:33:37.750 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:37 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' 2024-04-06T03:33:39.702 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:39 smithi060 ceph-mon[67116]: pgmap v422: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:39.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:39 smithi195 ceph-mon[53157]: pgmap v422: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:41.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:41 smithi195 ceph-mon[53157]: pgmap v423: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:41.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:41 smithi060 ceph-mon[67116]: pgmap v423: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:42.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:42 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:42.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:42 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:43.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:43 smithi195 ceph-mon[53157]: pgmap v424: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:43.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:43 smithi060 ceph-mon[67116]: pgmap v424: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:45.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:45 smithi195 ceph-mon[53157]: pgmap v425: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:45.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:45 smithi060 ceph-mon[67116]: pgmap v425: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:47.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:47 smithi195 ceph-mon[53157]: pgmap v426: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:47.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:47 smithi060 ceph-mon[67116]: pgmap v426: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:49.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:49 smithi195 ceph-mon[53157]: pgmap v427: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:49.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:49 smithi060 ceph-mon[67116]: pgmap v427: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:51.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:51 smithi195 ceph-mon[53157]: pgmap v428: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:51.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:51 smithi060 ceph-mon[67116]: pgmap v428: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:53.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:53 smithi195 ceph-mon[53157]: pgmap v429: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:53.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:53 smithi060 ceph-mon[67116]: pgmap v429: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:55.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:55 smithi195 ceph-mon[53157]: pgmap v430: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:55.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:55 smithi060 ceph-mon[67116]: pgmap v430: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:56.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch ps' 2024-04-06T03:33:57.499 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:57 smithi060 ceph-mon[67116]: pgmap v431: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:57.500 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:57 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:57.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:57 smithi195 ceph-mon[53157]: pgmap v431: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:33:57.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:57 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:alertmanager.smithi060 smithi060 *:9093,9094 running (109s) 87s ago 22m 15.0M - 0.25.0 c8568f914cd2 63aaf983a76b 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi060 smithi060 running (9m) 87s ago 22m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 60b81acfb782 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:crash.smithi195 smithi195 running (9m) 2m ago 21m 6660k - 19.0.0-2027-g9f192350 a5511c7deba9 ffc7d6eabb63 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:grafana.smithi060 smithi060 *:3000 running (93s) 87s ago 22m 48.7M - 9.4.12 2bacad6d85d8 c68ca5acf547 2024-04-06T03:33:58.599 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi060.hrodqu smithi060 running (3m) 87s ago 16m 33.9M - 19.0.0-2027-g9f192350 a5511c7deba9 426fa3f16968 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:mds.foofs.smithi195.dtvkez smithi195 running (3m) 2m ago 16m 12.6M - 19.0.0-2027-g9f192350 a5511c7deba9 84b07e74e933 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi060.jttwce smithi060 *:8443,9283,8765 running (12m) 87s ago 23m 523M - 19.0.0-2027-g9f192350 a5511c7deba9 6401e9d8ac51 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.smithi195.ultkot smithi195 *:8443,9283,8765 running (11m) 2m ago 21m 453M - 19.0.0-2027-g9f192350 a5511c7deba9 ddf24ab79a5f 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi060 smithi060 running (10m) 87s ago 24m 63.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 f24691574743 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:mon.smithi195 smithi195 running (10m) 2m ago 21m 50.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 3388bb07f1b2 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.0.0.smithi060.natxue smithi060 *:2049 running (2m) 87s ago 16m 74.6M - 5.7 a5511c7deba9 ba6fd13b0046 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:nfs.foo.1.0.smithi195.rdvmyy smithi195 *:2049 running (2m) 2m ago 16m 63.8M - 5.7 a5511c7deba9 9a320e957771 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi060 smithi060 *:9100 running (2m) 87s ago 22m 8262k - 1.5.0 0da6a335fe13 fcbd9c3db25f 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:node-exporter.smithi195 smithi195 *:9100 running (2m) 2m ago 21m 3263k - 1.5.0 0da6a335fe13 4c932fd5d566 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (8m) 87s ago 21m 400M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 dcafbe67e487 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi060 running (8m) 87s ago 20m 379M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 83a92d80097e 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi060 running (7m) 87s ago 20m 360M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 1a328081ebcc 2024-04-06T03:33:58.600 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (6m) 87s ago 19m 421M 1937M 19.0.0-2027-g9f192350 a5511c7deba9 5b5120f60184 2024-04-06T03:33:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi195 running (5m) 2m ago 19m 343M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 eb7e534914b1 2024-04-06T03:33:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi195 running (5m) 2m ago 18m 316M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 2b652aa44627 2024-04-06T03:33:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi195 running (4m) 2m ago 18m 291M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 f3003a577297 2024-04-06T03:33:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi195 running (3m) 2m ago 18m 279M 2704M 19.0.0-2027-g9f192350 a5511c7deba9 e07a448ef2a0 2024-04-06T03:33:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:prometheus.smithi060 smithi060 *:9095 running (2m) 87s ago 21m 46.8M - 2.43.0 a07b618ecd1d 664ec26b7b3e 2024-04-06T03:33:59.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch upgrade status' 2024-04-06T03:33:59.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:33:59 smithi060 ceph-mon[67116]: pgmap v432: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:33:59.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:33:59 smithi195 ceph-mon[53157]: pgmap v432: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:34:00.621 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:00 smithi060 ceph-mon[67116]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:00.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:00 smithi195 ceph-mon[53157]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:01.347 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:34:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "target_image": null, 2024-04-06T03:34:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "in_progress": false, 2024-04-06T03:34:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "which": "", 2024-04-06T03:34:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "services_complete": [], 2024-04-06T03:34:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "progress": null, 2024-04-06T03:34:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-04-06T03:34:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "is_paused": false 2024-04-06T03:34:01.349 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:34:01.697 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:01 smithi060 ceph-mon[67116]: pgmap v433: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:01.697 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:01 smithi060 ceph-mon[67116]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:01.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:01 smithi195 ceph-mon[53157]: pgmap v433: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:01.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:01 smithi195 ceph-mon[53157]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:01.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph health detail' 2024-04-06T03:34:02.560 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:02 smithi060 ceph-mon[67116]: pgmap v434: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:02.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:02 smithi195 ceph-mon[53157]: pgmap v434: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:04.113 INFO:teuthology.orchestra.run.smithi060.stdout:HEALTH_OK 2024-04-06T03:34:04.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions' 2024-04-06T03:34:04.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:04 smithi060 ceph-mon[67116]: pgmap v435: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:04.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:04 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/373305746' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:34:05.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:04 smithi195 ceph-mon[53157]: pgmap v435: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:05.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:04 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/373305746' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": { 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": { 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": { 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:34:06.828 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout: "overall": { 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 14 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:34:06.829 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-06T03:34:06.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:06 smithi195 ceph-mon[53157]: pgmap v436: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:06.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:06 smithi060 ceph-mon[67116]: pgmap v436: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:07.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-06T03:34:07.749 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:07 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3224338537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:07.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:07 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3224338537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:08.842 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:08 smithi060 ceph-mon[67116]: pgmap v437: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:34:08.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:08 smithi195 ceph-mon[53157]: pgmap v437: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:34:09.660 INFO:teuthology.orchestra.run.smithi060.stdout:true 2024-04-06T03:34:09.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:09 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/301932628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:09.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:09 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/301932628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:10.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-06T03:34:10.905 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:10 smithi060 ceph-mon[67116]: pgmap v438: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:10.973 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:10 smithi195 ceph-mon[53157]: pgmap v438: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:12.309 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)" 2024-04-06T03:34:12.980 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-04-06T03:34:12.981 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-04-06T03:34:12.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-06T03:34:13.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:12 smithi195 ceph-mon[53157]: pgmap v439: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:13.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:12 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/44915641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:13.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:12 smithi195 ceph-mon[53157]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:13.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:12 smithi060 ceph-mon[67116]: pgmap v439: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:13.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:12 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/44915641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T03:34:13.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:12 smithi060 ceph-mon[67116]: from='mgr.24613 172.21.15.60:0/3927576012' entity='mgr.smithi060.jttwce' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:15.013 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-06T03:34:15.013 INFO:teuthology.orchestra.run.smithi060.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-06T03:10:53.238905Z", "last_refresh": "2024-04-06T03:32:31.029509Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-06T03:10:46.906353Z", "last_refresh": "2024-04-06T03:31:37.856463Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-04-06T03:10:49.934828Z", "last_refresh": "2024-04-06T03:32:31.029664Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-06T03:17:21.184607Z", "last_refresh": "2024-04-06T03:31:37.857143Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-06T03:10:44.792800Z", "last_refresh": "2024-04-06T03:31:37.856570Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi060:172.21.15.60=smithi060", "smithi195:172.21.15.195=smithi195"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-06T03:11:37.279420Z", "last_refresh": "2024-04-06T03:31:37.856654Z", "running": 2, "size": 2}}, {"events": ["2024-04-06T03:33:36.983648Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-04-06T03:17:39.768389Z", "last_refresh": "2024-04-06T03:31:37.857234Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-06T03:10:51.152531Z", "last_refresh": "2024-04-06T03:31:37.856735Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", "last_refresh": "2024-04-06T03:31:37.856815Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-06T03:10:48.717338Z", "last_refresh": "2024-04-06T03:32:31.029815Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-06T03:34:15.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:14 smithi195 ceph-mon[53157]: pgmap v440: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:15.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:14 smithi060 ceph-mon[67116]: pgmap v440: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:15.605 INFO:tasks.cephadm:nfs.foo has 2/2 2024-04-06T03:34:15.606 DEBUG:teuthology.parallel:result is None 2024-04-06T03:34:15.606 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-06T03:34:15.623 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:34:15.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-04-06T03:34:15.650 INFO:teuthology.orchestra.run.smithi060.stderr:+ umount /mnt/foo 2024-04-06T03:34:15.661 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T03:34:15.670 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:34:15.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-04-06T03:34:16.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:15 smithi195 ceph-mon[53157]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:16.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:15 smithi060 ceph-mon[67116]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:16.999 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:16 smithi060 ceph-mon[67116]: pgmap v441: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:17.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:16 smithi195 ceph-mon[53157]: pgmap v441: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:17.822 INFO:teuthology.orchestra.run.smithi060.stdout: "foo" 2024-04-06T03:34:18.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs export ls foo --detailed' 2024-04-06T03:34:19.050 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:18 smithi060 ceph-mon[67116]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:19.050 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:18 smithi060 ceph-mon[67116]: pgmap v442: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:34:19.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:18 smithi195 ceph-mon[53157]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:19.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:18 smithi195 ceph-mon[53157]: pgmap v442: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-06T03:34:20.563 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "access_type": "RW", 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "clients": [], 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_id": "foo", 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "export_id": 1, 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "fsal": { 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "foofs", 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "CEPH", 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "user_id": "nfs.foo.1" 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "protocols": [ 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: 4 2024-04-06T03:34:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "pseudo": "/fake", 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "security_label": true, 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "squash": "none", 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "transports": [ 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "TCP" 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-06T03:34:20.565 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-04-06T03:34:21.162 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:20 smithi060 ceph-mon[67116]: pgmap v443: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:21.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'rados -p .nfs --all ls -' 2024-04-06T03:34:21.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:20 smithi195 ceph-mon[53157]: pgmap v443: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:22.220 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:21 smithi060 ceph-mon[67116]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:22.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:21 smithi195 ceph-mon[53157]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:34:23.040 INFO:teuthology.orchestra.run.smithi060.stdout:foo rec-0000000000000004:nfs.foo.1 2024-04-06T03:34:23.040 INFO:teuthology.orchestra.run.smithi060.stdout:foo grace 2024-04-06T03:34:23.040 INFO:teuthology.orchestra.run.smithi060.stdout:foo rec-0000000000000004:nfs.foo.0 2024-04-06T03:34:23.040 INFO:teuthology.orchestra.run.smithi060.stdout:foo conf-nfs.foo 2024-04-06T03:34:23.040 INFO:teuthology.orchestra.run.smithi060.stdout:foo export-1 2024-04-06T03:34:23.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:22 smithi195 ceph-mon[53157]: pgmap v444: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:23.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:22 smithi195 ceph-mon[53157]: mgrmap e42: smithi060.jttwce(active, since 11m), standbys: smithi195.ultkot 2024-04-06T03:34:23.249 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:22 smithi060 ceph-mon[67116]: pgmap v444: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:34:23.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:22 smithi060 ceph-mon[67116]: mgrmap e42: smithi060.jttwce(active, since 11m), standbys: smithi195.ultkot 2024-04-06T03:34:23.658 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T03:34:23.668 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:34:23.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 -- bash -c 'set -ex 2024-04-06T03:34:23.669 DEBUG:teuthology.orchestra.run.smithi060:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-04-06T03:34:23.669 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-04-06T03:34:25.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:24 smithi195 ceph-mon[53157]: pgmap v445: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T03:34:25.231 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:24 smithi060 ceph-mon[67116]: pgmap v445: 97 pgs: 97 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T03:34:25.430 INFO:teuthology.orchestra.run.smithi060.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-04-06T03:34:25.668 INFO:teuthology.orchestra.run.smithi060.stderr:+ [[ 6 -gt 2 ]] 2024-04-06T03:34:26.223 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:25 smithi195 ceph-mon[53157]: from='client.? 172.21.15.60:0/3382784181' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-06T03:34:26.250 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:25 smithi060 ceph-mon[67116]: from='client.? 172.21.15.60:0/3382784181' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-06T03:34:26.289 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-06T03:34:26.299 INFO:tasks.cephadm:Teardown begin 2024-04-06T03:34:26.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:34:26.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:34:26.365 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-06T03:34:26.365 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:34:26.384 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:34:26.423 INFO:tasks.cephadm:Stopping all daemons... 2024-04-06T03:34:26.423 INFO:tasks.cephadm.mon.smithi060:Stopping mon.smithi060... 2024-04-06T03:34:26.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060 2024-04-06T03:34:26.713 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:26 smithi060 systemd[1]: Stopping Ceph mon.smithi060 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:27.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:26 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[67112]: 2024-04-06T03:34:26.710+0000 7f8c57d35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi060 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:27.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:26 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060[67112]: 2024-04-06T03:34:26.710+0000 7f8c57d35700 -1 mon.smithi060@0(leader) e3 *** Got Signal Terminated *** 2024-04-06T03:34:27.000 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:26 smithi060 podman[112338]: 2024-04-06 03:34:26.843367344 +0000 UTC m=+0.225644371 container died f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-06T03:34:27.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:27 smithi060 podman[112338]: 2024-04-06 03:34:27.211660207 +0000 UTC m=+0.593937222 container cleanup f24691574743091ef043c636a45e5b02b31efb52ebec726dec888ac3ec5d9990 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:34:27.472 INFO:journalctl@ceph.mon.smithi060.smithi060.stdout:Apr 06 03:34:27 smithi060 bash[112338]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi060 2024-04-06T03:34:27.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi060.service' 2024-04-06T03:34:27.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:34:27.690 INFO:tasks.cephadm.mon.smithi060:Stopped mon.smithi060 2024-04-06T03:34:27.690 INFO:tasks.cephadm.mon.smithi195:Stopping mon.smithi195... 2024-04-06T03:34:27.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195 2024-04-06T03:34:27.974 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:27 smithi195 systemd[1]: Stopping Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:28.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195[53153]: 2024-04-06T03:34:27.995+0000 7ff3ceaea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi195 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:28.472 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195[53153]: 2024-04-06T03:34:27.995+0000 7ff3ceaea700 -1 mon.smithi195@1(peon) e3 *** Got Signal Terminated *** 2024-04-06T03:34:28.473 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:28 smithi195 podman[78570]: 2024-04-06 03:34:28.220322022 +0000 UTC m=+0.333690019 container died 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-06T03:34:28.723 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:28 smithi195 podman[78570]: 2024-04-06 03:34:28.662954023 +0000 UTC m=+0.776322012 container cleanup 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:34:28.724 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:28 smithi195 bash[78570]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195 2024-04-06T03:34:29.132 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:28 smithi195 podman[78616]: 2024-04-06 03:34:28.881136549 +0000 UTC m=+0.655219581 container remove 3388bb07f1b21ac57468c87f392aa929be8c7b07c6a4818463500fc8c4405436 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-mon-smithi195, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:34:29.132 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:29 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service: Deactivated successfully. 2024-04-06T03:34:29.132 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:29 smithi195 systemd[1]: Stopped Ceph mon.smithi195 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:34:29.132 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:Apr 06 03:34:29 smithi195 systemd[1]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service: Consumed 6.067s CPU time. 2024-04-06T03:34:29.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@mon.smithi195.service' 2024-04-06T03:34:29.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:34:29.184 INFO:tasks.cephadm.mon.smithi195:Stopped mon.smithi195 2024-04-06T03:34:29.184 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-06T03:34:29.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.0 2024-04-06T03:34:29.493 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:29 smithi060 systemd[1]: Stopping Ceph osd.0 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:29.750 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:29 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:34:29.491+0000 7f1075cea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:29.750 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:29 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:34:29.491+0000 7f1075cea700 -1 osd.0 158 *** Got signal Terminated *** 2024-04-06T03:34:29.750 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:29 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0[73632]: 2024-04-06T03:34:29.491+0000 7f1075cea700 -1 osd.0 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:34:35.219 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:34 smithi060 podman[112502]: 2024-04-06 03:34:34.931142251 +0000 UTC m=+5.544922749 container died dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-06T03:34:35.500 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:35 smithi060 podman[112502]: 2024-04-06 03:34:35.316583272 +0000 UTC m=+5.930363777 container cleanup dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-06T03:34:35.500 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:35 smithi060 bash[112502]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T03:34:35.945 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:35 smithi060 podman[112521]: 2024-04-06 03:34:35.576478286 +0000 UTC m=+0.642563610 container remove dcafbe67e487ef408c4c6cefba5d2fdd04dd58e3ac270977f0912ec3705c5642 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:34:36.240 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:35 smithi060 podman[112613]: 2024-04-06 03:34:35.848301502 +0000 UTC m=+0.019042553 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:34:36.240 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:35 smithi060 podman[112613]: 2024-04-06 03:34:35.969087284 +0000 UTC m=+0.139828309 container create 4f648b52447f0059a86d82d2d9c327e88ee27c074e6606a0d3c60025b52ec59d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9f19235) 2024-04-06T03:34:36.500 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:36 smithi060 podman[112613]: 2024-04-06 03:34:36.238877689 +0000 UTC m=+0.409618739 container init 4f648b52447f0059a86d82d2d9c327e88ee27c074e6606a0d3c60025b52ec59d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:34:36.500 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:36 smithi060 podman[112613]: 2024-04-06 03:34:36.2423956 +0000 UTC m=+0.413136633 container start 4f648b52447f0059a86d82d2d9c327e88ee27c074e6606a0d3c60025b52ec59d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-06T03:34:36.500 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:36 smithi060 podman[112613]: 2024-04-06 03:34:36.330478633 +0000 UTC m=+0.501219666 container attach 4f648b52447f0059a86d82d2d9c327e88ee27c074e6606a0d3c60025b52ec59d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:34:36.809 INFO:journalctl@ceph.osd.0.smithi060.stdout:Apr 06 03:34:36 smithi060 podman[112613]: 2024-04-06 03:34:36.53688986 +0000 UTC m=+0.707630905 container died 4f648b52447f0059a86d82d2d9c327e88ee27c074e6606a0d3c60025b52ec59d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:34:37.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-06T03:34:37.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:34:37.166 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-06T03:34:37.166 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-06T03:34:37.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.1 2024-04-06T03:34:37.500 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:37 smithi060 systemd[1]: Stopping Ceph osd.1 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:38.000 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:37 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:34:37.503+0000 7fbcc7744700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:38.000 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:37 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:34:37.503+0000 7fbcc7744700 -1 osd.1 158 *** Got signal Terminated *** 2024-04-06T03:34:38.000 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:37 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1[76150]: 2024-04-06T03:34:37.503+0000 7fbcc7744700 -1 osd.1 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:34:43.250 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:42 smithi060 podman[112718]: 2024-04-06 03:34:42.965078161 +0000 UTC m=+5.577957038 container died 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, io.buildah.version=1.33.5) 2024-04-06T03:34:43.606 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:43 smithi060 podman[112718]: 2024-04-06 03:34:43.33782057 +0000 UTC m=+5.950699439 container cleanup 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:34:43.607 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:43 smithi060 bash[112718]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T03:34:43.975 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:43 smithi060 podman[112737]: 2024-04-06 03:34:43.606034754 +0000 UTC m=+0.642114437 container remove 83a92d80097e488ad26367524b5646edd5b887dde9c56ac7af70ea2a21811e37 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-06T03:34:44.250 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:43 smithi060 podman[112827]: 2024-04-06 03:34:43.877432322 +0000 UTC m=+0.020085775 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:34:44.250 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:43 smithi060 podman[112827]: 2024-04-06 03:34:43.998799557 +0000 UTC m=+0.141453009 container create 5aed93d8acb2e4f92947ffb7ea615092f383d1c923217f76bdf0f5ccca30ece3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:34:44.250 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:44 smithi060 podman[112827]: 2024-04-06 03:34:44.218555211 +0000 UTC m=+0.361208664 container init 5aed93d8acb2e4f92947ffb7ea615092f383d1c923217f76bdf0f5ccca30ece3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:34:44.251 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:44 smithi060 podman[112827]: 2024-04-06 03:34:44.222388914 +0000 UTC m=+0.365042368 container start 5aed93d8acb2e4f92947ffb7ea615092f383d1c923217f76bdf0f5ccca30ece3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=squid-9f19235) 2024-04-06T03:34:44.750 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:44 smithi060 podman[112827]: 2024-04-06 03:34:44.301271962 +0000 UTC m=+0.443925432 container attach 5aed93d8acb2e4f92947ffb7ea615092f383d1c923217f76bdf0f5ccca30ece3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-06T03:34:44.750 INFO:journalctl@ceph.osd.1.smithi060.stdout:Apr 06 03:34:44 smithi060 podman[112827]: 2024-04-06 03:34:44.526238188 +0000 UTC m=+0.668891648 container died 5aed93d8acb2e4f92947ffb7ea615092f383d1c923217f76bdf0f5ccca30ece3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-06T03:34:45.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-06T03:34:45.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:34:45.198 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-06T03:34:45.198 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-06T03:34:45.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.2 2024-04-06T03:34:45.500 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:45 smithi060 systemd[1]: Stopping Ceph osd.2 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:46.000 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:45 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:34:45.566+0000 7f19209cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:46.000 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:45 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:34:45.566+0000 7f19209cd700 -1 osd.2 158 *** Got signal Terminated *** 2024-04-06T03:34:46.000 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:45 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2[78653]: 2024-04-06T03:34:45.566+0000 7f19209cd700 -1 osd.2 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:34:51.327 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:51 smithi060 podman[112933]: 2024-04-06 03:34:51.028971055 +0000 UTC m=+5.586671928 container died 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-06T03:34:51.710 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:51 smithi060 podman[112933]: 2024-04-06 03:34:51.424795903 +0000 UTC m=+5.982496759 container cleanup 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-06T03:34:51.710 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:51 smithi060 bash[112933]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T03:34:51.999 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:51 smithi060 podman[112982]: 2024-04-06 03:34:51.709692688 +0000 UTC m=+0.672754770 container remove 1a328081ebccaefa91baf2eb42488ae8cd2031300acbd46117ff55a368be5c83 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.build-date=20240403) 2024-04-06T03:34:52.431 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:51.990635134 +0000 UTC m=+0.018213091 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:34:52.431 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:52.110714307 +0000 UTC m=+0.138292242 container create fb40651f5a97d727f5ae4f0d34fce385dc19df33b113fd1666cf9e618d2920dd (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-06T03:34:52.431 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:52.330674936 +0000 UTC m=+0.358252874 container init fb40651f5a97d727f5ae4f0d34fce385dc19df33b113fd1666cf9e618d2920dd (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:34:52.432 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:52.333725426 +0000 UTC m=+0.361303366 container start fb40651f5a97d727f5ae4f0d34fce385dc19df33b113fd1666cf9e618d2920dd (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-9f19235, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-06T03:34:52.432 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:52.430176493 +0000 UTC m=+0.457754436 container attach fb40651f5a97d727f5ae4f0d34fce385dc19df33b113fd1666cf9e618d2920dd (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:34:53.000 INFO:journalctl@ceph.osd.2.smithi060.stdout:Apr 06 03:34:52 smithi060 podman[113076]: 2024-04-06 03:34:52.702704296 +0000 UTC m=+0.730282235 container died fb40651f5a97d727f5ae4f0d34fce385dc19df33b113fd1666cf9e618d2920dd (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T03:34:53.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-06T03:34:53.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:34:53.439 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-06T03:34:53.440 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-06T03:34:53.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.3 2024-04-06T03:34:53.750 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:53 smithi060 systemd[1]: Stopping Ceph osd.3 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:34:54.250 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:53 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:34:53.804+0000 7ff3e96b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:34:54.250 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:53 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:34:53.804+0000 7ff3e96b9700 -1 osd.3 158 *** Got signal Terminated *** 2024-04-06T03:34:54.250 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:53 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:34:53.804+0000 7ff3e96b9700 -1 osd.3 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:34:56.377 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:34:56 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:34:56.007+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:34:56.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:34:56 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:34:56.375+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:34:57.109 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:34:56 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:34:56.971+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:34:57.250 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:56 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:34:56.955+0000 7ff3e1ca5700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.500128+0000 front 2024-04-06T03:34:31.500044+0000 (oldest deadline 2024-04-06T03:34:56.800130+0000) 2024-04-06T03:34:57.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:34:57 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:34:57.347+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:34:57.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:34:57 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:34:57.107+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:34:58.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:34:57 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:34:57.936+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:34:58.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:34:58 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:34:58.094+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:34:58.249 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:57 smithi060 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3[81322]: 2024-04-06T03:34:57.914+0000 7ff3e1ca5700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.500128+0000 front 2024-04-06T03:34:31.500044+0000 (oldest deadline 2024-04-06T03:34:56.800130+0000) 2024-04-06T03:34:58.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:34:58 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:34:58.357+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:34:59.223 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:34:58 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:34:58.943+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:34:59.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:34:59 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:34:59.064+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:34:59.568 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:59 smithi060 podman[113180]: 2024-04-06 03:34:59.315359189 +0000 UTC m=+5.624138247 container died 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.build-date=20240403) 2024-04-06T03:34:59.724 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:34:59 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:34:59.308+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:34:59.972 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:59 smithi060 podman[113180]: 2024-04-06 03:34:59.670489989 +0000 UTC m=+5.979269043 container cleanup 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, ceph=True) 2024-04-06T03:34:59.973 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:59 smithi060 bash[113180]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T03:35:00.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:34:59 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:34:59.927+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:00.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:34:59 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:34:59.949+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:00.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:00.108+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:00.249 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:34:59 smithi060 podman[113209]: 2024-04-06 03:34:59.971917372 +0000 UTC m=+0.659149460 container remove 5b5120f601844c68919ff93bed92a4a23b955e6fb360d0056426ddccafe54beb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T03:35:00.719 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:00 smithi060 podman[113301]: 2024-04-06 03:35:00.20628154 +0000 UTC m=+0.020898498 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:35:00.720 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:00 smithi060 podman[113301]: 2024-04-06 03:35:00.324129246 +0000 UTC m=+0.138746189 container create ece4caae433920ee8512f1bf78a6899f4b87aa65d41e1b8b0030bb74eb7ea496 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-06T03:35:00.720 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:00 smithi060 podman[113301]: 2024-04-06 03:35:00.718756179 +0000 UTC m=+0.533373123 container init ece4caae433920ee8512f1bf78a6899f4b87aa65d41e1b8b0030bb74eb7ea496 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:35:00.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:00.336+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:01.000 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:00 smithi060 podman[113301]: 2024-04-06 03:35:00.722234127 +0000 UTC m=+0.536851071 container start ece4caae433920ee8512f1bf78a6899f4b87aa65d41e1b8b0030bb74eb7ea496 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T03:35:01.000 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:00 smithi060 podman[113301]: 2024-04-06 03:35:00.818849164 +0000 UTC m=+0.633466124 container attach ece4caae433920ee8512f1bf78a6899f4b87aa65d41e1b8b0030bb74eb7ea496 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5) 2024-04-06T03:35:01.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:00.913+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:01.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:00 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:00.975+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:01.225 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:01.066+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:01.314 INFO:journalctl@ceph.osd.3.smithi060.stdout:Apr 06 03:35:01 smithi060 podman[113301]: 2024-04-06 03:35:01.021431271 +0000 UTC m=+0.836048214 container died ece4caae433920ee8512f1bf78a6899f4b87aa65d41e1b8b0030bb74eb7ea496 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-06T03:35:01.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-06T03:35:01.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:35:01.638 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-06T03:35:01.638 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-06T03:35:01.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.4 2024-04-06T03:35:01.670 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:01.332+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:01.948 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:01 smithi195 systemd[1]: Stopping Ceph osd.4 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:35:02.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:01.956+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:02.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:02 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:02.024+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:02.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:01.946+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:02.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:01.972+0000 7f27810a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:35:02.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:01.972+0000 7f27810a0700 -1 osd.4 158 *** Got signal Terminated *** 2024-04-06T03:35:02.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:01 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:01.972+0000 7f27810a0700 -1 osd.4 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:35:02.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:02 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:02.312+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:03.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:02 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:02.909+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:03.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:02 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:02.952+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:03.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:03 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:03.024+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:03.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:03 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:03.308+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:04.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:03 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:03.991+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:04.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:03 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:03.926+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:04.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:03 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:03.903+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:04.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:04 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:04.273+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:05.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:04 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:04.910+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:05.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:04 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:04.992+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:05.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:04 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:04.970+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:05.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:05 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:05.261+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:06.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:06.001+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:06.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:06.010+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:30.674576+0000 front 2024-04-06T03:34:30.674491+0000 (oldest deadline 2024-04-06T03:34:55.974537+0000) 2024-04-06T03:35:06.224 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4[58080]: 2024-04-06T03:35:06.010+0000 7f277d694700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:40.576039+0000 front 2024-04-06T03:34:40.576113+0000 (oldest deadline 2024-04-06T03:35:05.276252+0000) 2024-04-06T03:35:06.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:05 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:05.866+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:06.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:05 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:05.866+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:06.724 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:06.262+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:06.724 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:06.262+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:07.222 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:07.050+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:07.222 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:07.050+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:07.222 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:06.836+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:07.222 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:06 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:06.836+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:07.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:07.219+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:07.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:07.219+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:07.775 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:07 smithi195 podman[78768]: 2024-04-06 03:35:07.49117776 +0000 UTC m=+5.650992651 container died eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-06T03:35:08.098 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:07 smithi195 podman[78768]: 2024-04-06 03:35:07.965607597 +0000 UTC m=+6.125422489 container cleanup eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-06T03:35:08.098 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:07 smithi195 bash[78768]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T03:35:08.098 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:07.860+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:08.098 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:07 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:07.860+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:08.474 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78787]: 2024-04-06 03:35:08.25078291 +0000 UTC m=+0.760226790 container remove eb7e534914b1ab7bf6289da7e9e96baee32bb09b1770caab35a933c4daf2d391 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5) 2024-04-06T03:35:08.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:08.096+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:08.475 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:08.096+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:08.475 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:08.253+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:08.475 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:08.253+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:08.827 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78911]: 2024-04-06 03:35:08.477029464 +0000 UTC m=+0.017133590 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:35:08.828 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78911]: 2024-04-06 03:35:08.576624104 +0000 UTC m=+0.116728207 container create 46400f12c44e633eae1ed39deb716921fced81fa085443e09495298614ec3a2e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:35:08.828 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78911]: 2024-04-06 03:35:08.787495046 +0000 UTC m=+0.327599145 container init 46400f12c44e633eae1ed39deb716921fced81fa085443e09495298614ec3a2e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9f19235) 2024-04-06T03:35:08.828 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78911]: 2024-04-06 03:35:08.790959626 +0000 UTC m=+0.331063734 container start 46400f12c44e633eae1ed39deb716921fced81fa085443e09495298614ec3a2e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:35:09.089 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:08 smithi195 podman[78911]: 2024-04-06 03:35:08.8710015 +0000 UTC m=+0.411105599 container attach 46400f12c44e633eae1ed39deb716921fced81fa085443e09495298614ec3a2e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-9f19235, io.buildah.version=1.33.5, ceph=True) 2024-04-06T03:35:09.090 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:08.825+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:09.090 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:08 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:08.825+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:09.343 INFO:journalctl@ceph.osd.4.smithi195.stdout:Apr 06 03:35:09 smithi195 podman[78911]: 2024-04-06 03:35:09.088967797 +0000 UTC m=+0.629071905 container died 46400f12c44e633eae1ed39deb716921fced81fa085443e09495298614ec3a2e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=) 2024-04-06T03:35:09.343 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:09.209+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:09.343 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:09.209+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:09.344 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:09.144+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:09.344 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:09.144+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:09.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-06T03:35:09.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:35:09.808 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-06T03:35:09.809 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-06T03:35:09.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.5 2024-04-06T03:35:09.974 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:09 smithi195 systemd[1]: Stopping Ceph osd.5 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:35:09.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:09.862+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:09.975 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:09 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:09.862+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:10.131+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:10.131+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:10.160+0000 7f91a7179700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:10.160+0000 7f91a7179700 -1 osd.5 158 *** Got signal Terminated *** 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:10.160+0000 7f91a7179700 -1 osd.5 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:35:10.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:10.237+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:10.475 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:10.237+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:11.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:10.870+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:11.224 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:10 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:10.870+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:11.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:11.106+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:11.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:11.106+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:11.723 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:11.260+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:11.724 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:11.260+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:12.153 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:11.821+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:12.153 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:11.821+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:12.153 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:11 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:11.821+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:12.473 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:12.244+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:12.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:12.244+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:12.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:12.151+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:12.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:12.151+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:12.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:12.151+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:13.191 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:12.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:13.191 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:12.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:13.191 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:12 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:12.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:13.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:13.250+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:13.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:13.250+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:13.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:13.189+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:13.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:13.189+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:13.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:13.189+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:14.189 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:13.781+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:14.189 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:13.781+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:14.189 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:13 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:13.781+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:14.300+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:33.677470+0000 front 2024-04-06T03:34:33.677495+0000 (oldest deadline 2024-04-06T03:34:55.377663+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:14.300+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.478665+0000 front 2024-04-06T03:34:39.478747+0000 (oldest deadline 2024-04-06T03:35:05.378785+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5[60325]: 2024-04-06T03:35:14.300+0000 7f919f765700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:47.679745+0000 front 2024-04-06T03:34:47.679847+0000 (oldest deadline 2024-04-06T03:35:13.579853+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:14.187+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:14.187+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:14.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:14.187+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:15.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:14.782+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:15.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:14.782+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:15.148 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:14 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:14.782+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:15.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:15.146+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:15.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:15.146+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:15.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:15.146+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:15.879 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:15 smithi195 podman[79027]: 2024-04-06 03:35:15.57741234 +0000 UTC m=+5.523627685 container died 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-04-06T03:35:15.880 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:15.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:15.880 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:15.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:15.880 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:15 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:15.816+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:16.224 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79027]: 2024-04-06 03:35:16.061535719 +0000 UTC m=+6.007751063 container cleanup 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403) 2024-04-06T03:35:16.224 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 bash[79027]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T03:35:16.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:16.106+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:16.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:16.106+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:16.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:16.106+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:16.598 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79045]: 2024-04-06 03:35:16.296323202 +0000 UTC m=+0.717706656 container remove 2b652aa446271028278744658be639b66ea1179fb38174846ac0d6845ce00c3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-06T03:35:16.925 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:16.840+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:16.925 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:16.840+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:16.925 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:16 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:16.840+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:16.926 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79135]: 2024-04-06 03:35:16.597035699 +0000 UTC m=+0.115125956 container create 876bfba7b3c380d7bf4b7df91d3e3a041d437fe8a1082435e62bf61199563051 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-06T03:35:16.926 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79135]: 2024-04-06 03:35:16.500133347 +0000 UTC m=+0.018223622 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:35:16.926 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79135]: 2024-04-06 03:35:16.824924327 +0000 UTC m=+0.343014592 container init 876bfba7b3c380d7bf4b7df91d3e3a041d437fe8a1082435e62bf61199563051 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-06T03:35:16.926 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79135]: 2024-04-06 03:35:16.828241881 +0000 UTC m=+0.346332136 container start 876bfba7b3c380d7bf4b7df91d3e3a041d437fe8a1082435e62bf61199563051 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-06T03:35:16.926 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:16 smithi195 podman[79135]: 2024-04-06 03:35:16.924502753 +0000 UTC m=+0.442593021 container attach 876bfba7b3c380d7bf4b7df91d3e3a041d437fe8a1082435e62bf61199563051 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_BRANCH=HEAD) 2024-04-06T03:35:17.224 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 06 03:35:17 smithi195 podman[79135]: 2024-04-06 03:35:17.117840763 +0000 UTC m=+0.635931011 container died 876bfba7b3c380d7bf4b7df91d3e3a041d437fe8a1082435e62bf61199563051 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-06T03:35:17.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:17.145+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:17.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:17.145+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:17.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:17.145+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:17.820 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:17.796+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:17.821 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:17.796+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:17.821 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:17 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:17.796+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:17.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-06T03:35:17.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:35:17.887 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-06T03:35:17.887 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-06T03:35:17.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.6 2024-04-06T03:35:18.163 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:17 smithi195 systemd[1]: Stopping Ceph osd.6 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:18.161+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:18.161+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:18.161+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.214+0000 7fe777f3c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.214+0000 7fe777f3c700 -1 osd.6 158 *** Got signal Terminated *** 2024-04-06T03:35:18.474 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.214+0000 7fe777f3c700 -1 osd.6 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:35:19.154 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.753+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:19.154 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.753+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:19.154 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:18 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:18.753+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:19.473 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:19.153+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:19.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:19.153+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:19.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:19.153+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:20.161 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:19.789+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:20.161 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:19.789+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:20.161 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:19 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:19.789+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:20.473 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:20.160+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:20.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:20.160+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:20.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:20.160+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:21.131 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:20.814+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:21.132 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:20.814+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:21.132 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:20.814+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:21.132 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:20 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:20.814+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.650985+0000 front 2024-04-06T03:34:57.651062+0000 (oldest deadline 2024-04-06T03:35:19.951040+0000) 2024-04-06T03:35:21.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:21.130+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:21.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:21.130+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:21.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:21.130+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:22.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:21.773+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:22.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:21.773+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:22.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:21.773+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:22.147 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:21 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:21.773+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.650985+0000 front 2024-04-06T03:34:57.651062+0000 (oldest deadline 2024-04-06T03:35:19.951040+0000) 2024-04-06T03:35:22.473 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:22.144+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:22.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:22.144+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:22.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:22.144+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:23.177 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:22.740+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:31.947875+0000 front 2024-04-06T03:34:31.947895+0000 (oldest deadline 2024-04-06T03:34:59.548287+0000) 2024-04-06T03:35:23.177 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:22.740+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:41.849252+0000 front 2024-04-06T03:34:41.849183+0000 (oldest deadline 2024-04-06T03:35:05.349220+0000) 2024-04-06T03:35:23.178 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:22.740+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:45.349673+0000 front 2024-04-06T03:34:45.349718+0000 (oldest deadline 2024-04-06T03:35:11.249604+0000) 2024-04-06T03:35:23.178 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:22 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6[62568]: 2024-04-06T03:35:22.740+0000 7fe774530700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.650985+0000 front 2024-04-06T03:34:57.651062+0000 (oldest deadline 2024-04-06T03:35:19.951040+0000) 2024-04-06T03:35:23.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:23.176+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:23.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:23.176+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:23.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:23 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:23.176+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:23.974 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:23 smithi195 podman[79239]: 2024-04-06 03:35:23.673020559 +0000 UTC m=+5.568965345 container died f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS) 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79239]: 2024-04-06 03:35:24.072738411 +0000 UTC m=+5.968683208 container cleanup f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 bash[79239]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:24 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:24.225+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:24 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:24.225+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:24 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:24.225+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:24.333 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:24 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:24.225+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:24.687 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79258]: 2024-04-06 03:35:24.332275774 +0000 UTC m=+0.660304850 container remove f3003a577297aff9c114093f555ec484bea157e0e41fc079d48ad62099d841bb (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235) 2024-04-06T03:35:24.970 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79348]: 2024-04-06 03:35:24.589098958 +0000 UTC m=+0.017538533 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:35:24.970 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79348]: 2024-04-06 03:35:24.691600633 +0000 UTC m=+0.120040251 container create 70319964a30bfd76ccc202e685b3e89fe9dd7dcb72910ee93dd6ff8d014f6985 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-06T03:35:24.970 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79348]: 2024-04-06 03:35:24.894621832 +0000 UTC m=+0.323061408 container init 70319964a30bfd76ccc202e685b3e89fe9dd7dcb72910ee93dd6ff8d014f6985 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T03:35:24.970 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79348]: 2024-04-06 03:35:24.89789373 +0000 UTC m=+0.326333308 container start 70319964a30bfd76ccc202e685b3e89fe9dd7dcb72910ee93dd6ff8d014f6985 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-06T03:35:24.970 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:24 smithi195 podman[79348]: 2024-04-06 03:35:24.96909346 +0000 UTC m=+0.397533062 container attach 70319964a30bfd76ccc202e685b3e89fe9dd7dcb72910ee93dd6ff8d014f6985 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-06T03:35:25.223 INFO:journalctl@ceph.osd.6.smithi195.stdout:Apr 06 03:35:25 smithi195 podman[79348]: 2024-04-06 03:35:25.18432857 +0000 UTC m=+0.612768147 container died 70319964a30bfd76ccc202e685b3e89fe9dd7dcb72910ee93dd6ff8d014f6985 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-6-deactivate, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-06T03:35:25.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:25 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:25.221+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:25.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:25 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:25.221+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:25.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:25 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:25.221+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:25.224 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:25 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:25.221+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:25.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-06T03:35:25.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:35:25.862 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-06T03:35:25.862 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-06T03:35:25.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.7 2024-04-06T03:35:26.186 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:25 smithi195 systemd[1]: Stopping Ceph osd.7 for 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:35:26.473 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.183+0000 7f80b774f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.183+0000 7f80b774f700 -1 osd.7 158 *** Got signal Terminated *** 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.183+0000 7f80b774f700 -1 osd.7 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.196+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.196+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.196+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:26.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:26 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:26.196+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:27.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:27.224+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:27.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:27.224+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:27.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:27.224+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:27.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:27 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:27.224+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:28.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:28 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:28.209+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:28.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:28 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:28.209+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:28.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:28 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:28.209+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:28.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:28 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:28.209+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:29.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:29.233+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:29.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:29.233+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:29.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:29.233+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:29.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:29.233+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:29.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:29 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:29.233+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-04-06T03:35:06.098784+0000 front 2024-04-06T03:35:06.098825+0000 (oldest deadline 2024-04-06T03:35:28.398775+0000) 2024-04-06T03:35:30.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:30.204+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-04-06T03:34:32.894504+0000 front 2024-04-06T03:34:32.894533+0000 (oldest deadline 2024-04-06T03:34:56.994290+0000) 2024-04-06T03:35:30.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:30.204+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6814 osd.1 since back 2024-04-06T03:34:39.795745+0000 front 2024-04-06T03:34:39.795731+0000 (oldest deadline 2024-04-06T03:35:06.796280+0000) 2024-04-06T03:35:30.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:30.204+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6822 osd.2 since back 2024-04-06T03:34:46.796713+0000 front 2024-04-06T03:34:46.796853+0000 (oldest deadline 2024-04-06T03:35:11.496794+0000) 2024-04-06T03:35:30.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:30.204+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.60:6830 osd.3 since back 2024-04-06T03:34:57.298034+0000 front 2024-04-06T03:34:57.297999+0000 (oldest deadline 2024-04-06T03:35:23.198046+0000) 2024-04-06T03:35:30.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:30 smithi195 ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7[64814]: 2024-04-06T03:35:30.204+0000 7f80b3d43700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-04-06T03:35:06.098784+0000 front 2024-04-06T03:35:06.098825+0000 (oldest deadline 2024-04-06T03:35:28.398775+0000) 2024-04-06T03:35:32.110 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:31 smithi195 podman[79454]: 2024-04-06 03:35:31.815752435 +0000 UTC m=+5.733991600 container died e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:35:32.428 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79454]: 2024-04-06 03:35:32.208558567 +0000 UTC m=+6.126797738 container cleanup e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-06T03:35:32.428 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 bash[79454]: ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T03:35:32.724 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79515]: 2024-04-06 03:35:32.426611953 +0000 UTC m=+0.614437101 container remove e07a448ef2a0dbe9a55fa197e04a84208f844132710a774d22d0f042b2893ff1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, ceph=True) 2024-04-06T03:35:33.047 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79607]: 2024-04-06 03:35:32.645457771 +0000 UTC m=+0.017880474 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:35:33.047 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79607]: 2024-04-06 03:35:32.743817513 +0000 UTC m=+0.116240216 container create b76b9f024559bca91f9599dcaf415db322ef20ce0137e60e3cab897ead7c4e23 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:35:33.047 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79607]: 2024-04-06 03:35:32.962706558 +0000 UTC m=+0.335129262 container init b76b9f024559bca91f9599dcaf415db322ef20ce0137e60e3cab897ead7c4e23 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-06T03:35:33.047 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:32 smithi195 podman[79607]: 2024-04-06 03:35:32.965846428 +0000 UTC m=+0.338269131 container start b76b9f024559bca91f9599dcaf415db322ef20ce0137e60e3cab897ead7c4e23 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, ceph=True, CEPH_POINT_RELEASE=) 2024-04-06T03:35:33.047 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:33 smithi195 podman[79607]: 2024-04-06 03:35:33.046071577 +0000 UTC m=+0.418494281 container attach b76b9f024559bca91f9599dcaf415db322ef20ce0137e60e3cab897ead7c4e23 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T03:35:33.474 INFO:journalctl@ceph.osd.7.smithi195.stdout:Apr 06 03:35:33 smithi195 podman[79607]: 2024-04-06 03:35:33.258579912 +0000 UTC m=+0.631002619 container died b76b9f024559bca91f9599dcaf415db322ef20ce0137e60e3cab897ead7c4e23 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T03:35:33.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ee7dd52-f3c3-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-06T03:35:33.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:35:33.893 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-06T03:35:33.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T03:38:30.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T03:38:51.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:38:51.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:38:51.648 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-06T03:38:51.651 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi060/crash 2024-04-06T03:38:51.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T03:38:51.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T03:38:51.699 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2024-04-06T03:38:51.701 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi195/crash 2024-04-06T03:38:51.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T03:38:51.736 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T03:38:51.736 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-04-06T03:38:51.737 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-06T03:38:51.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T03:38:51.771 INFO:teuthology.orchestra.run.smithi060.stdout:2024-04-06T03:22:06.681997+0000 mon.smithi060 (mon.0) 866 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:38:51.771 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-06T03:38:51.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[SEC\]' /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T03:38:51.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]' /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T03:38:51.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[WRN\]' /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T03:38:51.983 INFO:teuthology.orchestra.run.smithi060.stdout:2024-04-06T03:22:06.681997+0000 mon.smithi060 (mon.0) 866 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T03:38:51.983 INFO:tasks.cephadm:Compressing logs... 2024-04-06T03:38:51.983 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T03:38:52.026 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T03:38:52.051 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:: No such file or directory 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi060.jttwce.log 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi060.log 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi060.jttwce.log: /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-06T03:38:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi060.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-06T03:38:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-06T03:38:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T03:38:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T03:38:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T03:38:52.055 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-06T03:38:52.056 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T03:38:52.056 INFO:teuthology.orchestra.run.smithi195.stderr:: No such file or directory 2024-04-06T03:38:52.056 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log 2024-04-06T03:38:52.056 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi195.log 2024-04-06T03:38:52.057 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi195.dtvkez.log 2024-04-06T03:38:52.057 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi195.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T03:38:52.058 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log: 91.5% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T03:38:52.061 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-06T03:38:52.061 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi195.dtvkez.log: /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log 2024-04-06T03:38:52.061 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi195.dtvkez.log.gz 2024-04-06T03:38:52.061 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-06T03:38:52.061 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log: 91.6% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr: 87.1% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T03:38:52.062 INFO:teuthology.orchestra.run.smithi195.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T03:38:52.063 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T03:38:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T03:38:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi060.hrodqu.log 2024-04-06T03:38:52.066 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi195.ultkot.log 2024-04-06T03:38:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log: /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi060.hrodqu.log: 89.2% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi060.hrodqu.log.gz 2024-04-06T03:38:52.079 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T03:38:52.082 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log: /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi195.ultkot.log: 91.5% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T03:38:52.089 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T03:38:52.119 INFO:teuthology.orchestra.run.smithi195.stderr: 89.8% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi195.ultkot.log.gz 2024-04-06T03:38:52.450 INFO:teuthology.orchestra.run.smithi195.stderr: 92.7% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi195.log.gz 2024-04-06T03:38:52.466 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.smithi060.jttwce.log.gz 2024-04-06T03:38:54.106 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.smithi060.log.gz 2024-04-06T03:39:04.463 INFO:teuthology.orchestra.run.smithi060.stderr: 94.5% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-06T03:39:08.012 INFO:teuthology.orchestra.run.smithi195.stderr: 93.9% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-06T03:39:09.279 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-06T03:39:10.916 INFO:teuthology.orchestra.run.smithi060.stderr: 94.1% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-06T03:39:11.461 INFO:teuthology.orchestra.run.smithi060.stderr: 94.2% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-06T03:39:12.909 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-06T03:39:13.282 INFO:teuthology.orchestra.run.smithi060.stderr: 94.1% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-06T03:39:13.284 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-06T03:39:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m21.246s 2024-04-06T03:39:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:user 1m11.746s 2024-04-06T03:39:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m2.584s 2024-04-06T03:39:17.571 INFO:teuthology.orchestra.run.smithi195.stderr: 94.1% -- replaced with /var/log/ceph/0ee7dd52-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-06T03:39:17.573 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-04-06T03:39:17.573 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m25.532s 2024-04-06T03:39:17.573 INFO:teuthology.orchestra.run.smithi195.stderr:user 1m16.770s 2024-04-06T03:39:17.573 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m2.959s 2024-04-06T03:39:17.573 INFO:tasks.cephadm:Archiving logs... 2024-04-06T03:39:17.573 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi060/log 2024-04-06T03:39:17.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T03:39:27.473 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi195/log 2024-04-06T03:39:27.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T03:39:38.309 INFO:tasks.cephadm:Removing cluster... 2024-04-06T03:39:38.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 --force 2024-04-06T03:39:38.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ee7dd52-f3c3-11ee-b649-cb9ed24678a4 --force 2024-04-06T03:39:39.081 INFO:tasks.cephadm:Removing cephadm ... 2024-04-06T03:39:39.082 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T03:39:39.097 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T03:39:39.112 INFO:tasks.cephadm:Teardown complete 2024-04-06T03:39:39.113 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T03:39:39.123 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T03:39:39.123 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:39:39.140 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:39:39.151 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 -417us[ -417us] +/- 88ms 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -130us[ -129us] +/- 50ms 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +599us[ +599us] +/- 82ms 2024-04-06T03:39:39.157 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:39:39.166 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 -391us[ -391us] +/- 88ms 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 65 -120us[ -87us] +/- 65ms 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 65 +596us[ +630us] +/- 81ms 2024-04-06T03:39:39.170 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:39:39.170 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T03:39:39.208 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T03:39:39.209 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T03:39:39.219 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T03:39:39.304 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T03:39:39.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T03:39:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T03:39:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.443 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T03:39:39.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T03:39:39.463 INFO:teuthology.task.internal:Duration was 2087.426389 seconds 2024-04-06T03:39:39.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T03:39:39.473 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T03:39:39.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T03:39:39.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T03:39:39.516 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T03:39:39.526 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T03:39:39.789 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T03:39:39.789 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-04-06T03:39:39.790 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T03:39:39.850 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-04-06T03:39:39.850 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T03:39:39.875 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T03:39:39.875 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T03:39:39.893 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T03:39:39.944 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T03:39:39.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T03:39:39.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T03:39:53.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T03:39:53.862 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T03:39:53.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T03:39:53.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T03:39:53.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T03:39:53.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T03:39:53.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T03:39:53.958 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-04-06T03:39:53.990 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-04-06T03:39:54.028 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T03:39:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.041 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T03:39:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T03:39:54.095 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T03:39:54.096 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi060 2024-04-06T03:39:54.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T03:39:54.160 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915/remote/smithi195 2024-04-06T03:39:54.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T03:39:54.203 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T03:39:54.203 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T03:39:54.205 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T03:39:54.251 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T03:39:54.263 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T03:39:54.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T03:39:54.334 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T03:39:54.334 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T03:39:54.337 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T03:39:54.348 INFO:teuthology.orchestra.run.smithi060.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 03:39 /home/ubuntu/cephtest 2024-04-06T03:39:54.352 INFO:teuthology.orchestra.run.smithi195.stdout: 265402 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 03:39 /home/ubuntu/cephtest 2024-04-06T03:39:54.353 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T03:39:54.365 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T03:39:54.387 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T03:39:54.416 DEBUG:teuthology.nuke:shortname: smithi060 2024-04-06T03:39:54.416 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T03:39:54.438 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:56:38.975483', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGJE9dFfyLhLYJZCyfNuJuhCTRz2BVsRNYM1v8ZM+nydf1kdjmh0kncdf30Sly5H99ubXZyoJ32SaNyikF2QQ0='} 2024-04-06T03:39:54.441 DEBUG:teuthology.nuke:shortname: smithi195 2024-04-06T03:39:54.441 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T03:39:54.463 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:56:38.976930', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJwL8gk2Il6mUZCvxSVovtI4ZIlAuhXbjaSBaebCdfqB7YXK2MUM6vbQ8QrDI/Rc6DJwGhJbpvFUvxrVpf1uEY='} 2024-04-06T03:39:54.492 INFO:teuthology.orchestra.console.smithi060:Power off 2024-04-06T03:39:54.492 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T03:39:54.516 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-04-06T03:39:54.516 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:39:54.532 INFO:teuthology.orchestra.console.smithi195:Power off 2024-04-06T03:39:54.532 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T03:39:54.541 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-04-06T03:39:54.554 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-04-06T03:39:54.555 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:39:54.577 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-04-06T03:39:58.542 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:39:58.578 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:39:58.652 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-04-06T03:39:58.690 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-04-06T03:40:02.656 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:40:02.691 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:40:02.766 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-04-06T03:40:02.800 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-04-06T03:40:02.868 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-04-06T03:40:02.901 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-04-06T03:40:03.006 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2087.4263892173767 failure_reason: '"2024-04-06T03:22:06.681997+0000 mon.smithi060 (mon.0) 866 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-06T03:40:03.006 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T03:40:03.087 INFO:teuthology.run:FAIL